Skip to content

perf(pgsql): batch schema fetching for changed tables (#165, #184) - #183

Merged
jasdeepkhalsa merged 5 commits into
masterfrom
claude/dbdiff-supaforge-roadmap-syfheh
Aug 10, 2026
Merged

jasdeepkhalsa merged 5 commits into
masterfrom
claude/dbdiff-supaforge-roadmap-syfheh

Conversation

@jasdeepkhalsa

@jasdeepkhalsa jasdeepkhalsa commented Aug 10, 2026

Copy link
Copy Markdown
Member

Closes #165. Closes #184.

Makes schema diffing cost a constant number of round-trips instead of one set per table, then fixes the query that was absorbing most of that gain. Along the way this fixes a constraint-name collision bug and adds test coverage for the two performance passes, neither of which had any.

1. Batch schema fetching (#165)

The pre-scan in #180 skips tables that are already identical. The tables that genuinely differ still needed 8 queries per table per side. This loads them all together instead.

  • New BulkSchemaAdapterInterface with a single getBulkTableSchema() method. It is a separate opt-in interface, so DBAdapterInterface is untouched and MySQL/SQLite need no changes — MySQL already resolves a table in ~2 queries, and SQLite is a local file with no network latency.
  • PostgresAdapter implements it: 7 fixed queries per side regardless of table count. getTableSchema() and getCreateStatement() now delegate to it.
  • DBSchema::getDiff() partitions common tables by hash, then bulk-fetches only the changed ones.
  • TableSchema::getDiff() accepts pre-fetched schemas, falling back to per-table queries when none are supplied.

For a 300-table database with 7 real changes: 16 queries total, against roughly 4,800 with per-table fetching.

Also fixes a duplicate query — the named NOT NULL lookup previously ran twice per table, once from fetchColumns() and once from fetchConstraints().

2. pg_catalog rewrite of the constraint query (#184)

Timing the seven batch queries individually showed one was 99.2% of the whole fetch — information_schema.table_constraints joined to key_column_usage, referential_constraints and constraint_column_usage. Those views wrap the catalogs in per-row privilege checks, which stops the planner pushing table_name IN (...) down through them.

Reading pg_constraint directly, with unnest(conkey) WITH ORDINALITY for the column list:

tables in batch before after
100 4013 ms 40 ms
250 9721 ms 83 ms
500 19920 ms 157 ms
1000 39246 ms 324 ms

Roughly 120x, taking per-table cost from ~39 ms to ~0.3 ms.

Output is unchanged — the CASE arms reproduce exactly what the information_schema views emit, including Postgres mapping simple match to NONE rather than SIMPLE. Incidentally, the column ordinal is now explicit rather than relying on key_column_usage ordering, and pg_catalog does not omit constraints the connecting role lacks privileges on.

3. Constraint-name collision fix

Found while adding tests. The constraint query joined pg_constraint on conname alone, but Postgres scopes constraint names per table, not per schema. A constraint on one table matched a same-named constraint on another, duplicating the row and producing corrupt DDL:

old (name-only join):  users | shared_name | ref     <- duplicated
                       users | shared_name | ref
new (table-scoped):    users | shared_name | ref

That duplicate became FOREIGN KEY ("ref", "ref"). Reproduced on Postgres 16 with two tables sharing a constraint name. Grouped constraint columns are now keyed by name so any join fan-out collapses — an N-column FK referencing an N-column key produces N×N rows. Constraints buildConstraintDef() cannot render are dropped rather than stored as nulls in the diff map.

Shared helpers

MySQLAdapter and PostgresAdapter both sit on the 20-method SonarCloud ceiling, so two duplicated snippets moved to DBDiff\DB\Support\QueryHelper: IN-clause placeholder building (PostgresAdapter, StreamingMergeDiff) and table-map narrowing (getSchemaHashMap on both adapters).

Tests

Both performance passes had shipped with no coverage. This adds 91 tests:

  • QueryHelperTest — placeholder generation and table narrowing.
  • PostgresBulkSchemaTest — assembly of columns, indexes and constraints from pre-fetched rows, concentrating on the cross-table isolation that single-table fetching used to provide for free.
  • SchemaBatchFetchTestDBSchema wiring, asserting the negatives: unchanged tables are never fetched, changed tables are batched exactly once per side, non-bulk adapters still fall back per table.
  • BulkSchemaPostgresTest — live-server suite asserting the batch path returns exactly what the per-table path returns, that the query budget stays at 7 per side regardless of table count, that schema hashes track schema but not data, and that every referential-action mapping renders correctly. It builds its own database and compares the two code paths against each other, so it needs no per-version golden files.

Verified against a live Postgres 16: 788 tests pass, including the golden-file end-to-end migration test, which still matches byte for byte. The new suite is registered in the Postgres testsuite, so CI runs it across PG 14–18.

Docs

The README documented neither performance pass. Adds a Schema Diff Performance section with the per-driver matrix and the round-trip maths. Separately, --allow-destructive from #174 had shipped with no README coverage at all — the flag was missing from the options table and the blocking behaviour was undocumented — so that gets a section too.

Not included

Chunking the IN (...) list, measured and rejected. At 1000 tables it makes under 1% difference (39246 ms vs 38908 ms, within noise), peak memory is ~4 MB, and padding the parameter list to 65000 while holding result size fixed adds only ~0.6 s. The 65535 bind-parameter ceiling is real and fails cleanly, but requires more than 65535 changed tables in a single diff. Rationale recorded in #184.

claude added 3 commits August 10, 2026 17:03
Replace O(N) per-table queries with O(1) batch queries when tables
differ between source and target. For N changed tables, PostgresAdapter
now runs 7 fixed queries per side (14 total) regardless of N, down from
8 per table per side (16N total).

Key changes:
- Add BulkSchemaAdapterInterface with getBulkTableSchema() contract
- PostgresAdapter implements the new interface; getBulkTableSchema() runs
  7 IN-clause batch queries and assembles results via assembleColumns(),
  assembleIndexes(), assembleConstraints()
- getTableSchema() and getCreateStatement() delegate to getBulkTableSchema()
- DBSchema.getDiff() bulk-fetches schemas for all changed tables before
  the diff loop when the adapter implements BulkSchemaAdapterInterface
- TableSchema.getDiff() accepts optional pre-fetched schemas to avoid
  redundant DB queries
- Bonus: fixes the duplicate named-NOT-NULL-constraint query that previously
  ran twice per table (once from fetchColumns, once from fetchConstraints)

MySQL and SQLite adapters are unaffected (MySQL already runs ~2 queries
per table; SQLite is always local). The BulkSchemaAdapterInterface is a
separate opt-in interface so DBAdapterInterface is unchanged.
…collision

Audit of the two most recent performance features (#179/#180 pre-scan and
#165 batch fetch) found both had shipped with no test coverage, and turned
up a correctness bug in the batched constraint query.

Correctness
- Scope the pg_constraint join to the owning table. Postgres scopes
  constraint names per table, not per schema, so joining on conname alone
  matched constraints belonging to other tables. Verified on PG16: two
  tables sharing a constraint name returned a duplicated row, which the
  grouping step turned into FOREIGN KEY ("ref", "ref").
- Key grouped constraint columns by name so any join fan-out collapses
  (an N-column FK referencing an N-column key produces N x N rows).
- Drop constraints buildConstraintDef cannot render instead of storing
  nulls in the diff map; restores behaviour lost with array_filter().
- Correct the BulkSchemaAdapterInterface docblock, which described a
  fallback the implementation does not have.

Shared helpers
- Extract DBDiff\DB\Support\QueryHelper for the two duplicated snippets:
  IN-clause placeholder building (PostgresAdapter, StreamingMergeDiff) and
  table-map narrowing (getSchemaHashMap on both MySQL and Postgres). Both
  adapters sit on the 20-method SonarCloud ceiling, so shared behaviour has
  to live outside them.

Tests (+90)
- QueryHelperTest: placeholder and narrowing behaviour.
- PostgresBulkSchemaTest: assembly of columns, indexes and constraints from
  pre-fetched rows, focused on the cross-table isolation that single-table
  fetching used to provide for free.
- SchemaBatchFetchTest: DBSchema wiring — unchanged tables are never
  fetched, changed tables are batched exactly once per side, and adapters
  without bulk support still fall back per table.
- BulkSchemaPostgresTest: live-server suite asserting the batch path
  returns exactly what the per-table path returns, that the query budget
  stays at 7 per side regardless of table count, and that schema hashes
  track schema changes but not data writes. Compares the two code paths
  against each other, so it needs no per-version golden files.

Docs
- Document both passes in the README with the per-driver matrix and the
  round-trip maths behind them.
The linter and its CLI flag shipped in #174 with no README coverage: the
flag was missing from the options table and the blocking behaviour was
undocumented, so a first run against a database with dropped tables fails
with no explanation in the docs.

Adds a Destructive Change Protection section covering the error/warning
split, the possible-rename downgrade, and the allowDestructive config key,
and lists the flag in the diff options table.
@github-actions github-actions Bot added mysql Related to Mysql php Pull requests that update php code sqlite Related to Sqlite labels Aug 10, 2026
claude added 2 commits August 10, 2026 18:20
… (S3776)

DBSchema::getDiff (19 -> ~10)
Extract the two blocks added for #165 into their own methods:
selectTablesNeedingDiff() for the pre-scan partition and bulkFetchSchemas()
for the batched load. getDiff() reads as a sequence of phases again, and
each helper is well under the limit on its own.

PostgresAdapter::assembleConstraints (18 -> ~12)
This class sits on the 20-method ceiling (S2166), so the complexity had to
come out in place rather than into a helper. Both nested loop pairs are
flattened:

- FK/UNIQUE/PK rows group under one table+constraint key joined by NUL,
  which no Postgres identifier can contain, so the emit loop is single-level
  and rebuilds the map from each row's own table_name/constraint_name.
- Named NOT NULL constraints are collected as a flat list instead of a
  [table][name] map. Each row already carries both names, and the query
  yields exactly one row per constraint, so nothing needed the nesting.

Output is unchanged: assembled constraints still come out grouped by table
in the order the ORDER BY produces. Adds a test that named NOT NULL
constraints sharing a name across tables stay scoped to their own table,
matching the existing coverage for the other constraint kinds.
Timing the seven batch queries individually showed one of them was 99% of
the entire schema fetch: the information_schema.table_constraints join to
key_column_usage, referential_constraints and constraint_column_usage.

Those views wrap the catalogs in per-row privilege checks, which stops the
planner pushing `table_name IN (...)` down through them, so they are largely
materialised before the filter applies. Reading pg_constraint directly, with
unnest(conkey) WITH ORDINALITY for the column list, returns the same rows
without any of that.

Measured on Postgres 16 against a 1000-table database, batch fetching:

    tables    before     after
       100    4013 ms     40 ms
       250    9721 ms     83 ms
       500   19920 ms    157 ms
      1000   39246 ms    324 ms

Roughly 120x, taking per-table cost from ~39 ms to ~0.3 ms. #165 removed the
per-table round-trips; this removes what was absorbing most of that gain.

Output is unchanged. The CASE arms reproduce exactly what the
information_schema views emit, including Postgres mapping simple match to
'NONE' rather than 'SIMPLE'. Verified by running both queries over a schema
covering composite PK/UNIQUE/FK, all five referential actions, MATCH FULL,
both deferrable modes and NOT VALID, then comparing what the assembly step
consumes: identical across all 15 constraints. The golden-file end-to-end
migration test also still matches byte for byte.

Adds a live-server test pinning every arm of the mapping, since a wrong
catalog code would silently emit the wrong referential action into a
migration rather than failing loudly.

Two incidental improvements: the constraint column ordinal is now explicit
instead of relying on key_column_usage ordering, and pg_catalog does not
omit constraints the connecting role lacks privileges on.
@jasdeepkhalsa jasdeepkhalsa changed the title feat: batch schema fetching for changed tables (issue #165) perf(pgsql): batch schema fetching for changed tables (#165, #184) Aug 10, 2026
@sonarqubecloud

Copy link
Copy Markdown

@jasdeepkhalsa
jasdeepkhalsa merged commit d96170d into master Aug 10, 2026
67 checks passed
jasdeepkhalsa added a commit that referenced this pull request Aug 23, 2026
The branch was cut from a stale master and missed 20 commits, including the
bulk schema fetch (#183), reading constraints from pg_catalog (#184) and the
cognitive-complexity refactor. All four fixes are re-applied on top of that
work rather than around it:

  - the duplicate PRIMARY KEY is removed from the rebuilt CREATE TABLE, which
    master still emitted via getPrimaryKey() alongside the named constraint;

  - sequence ownership is resolved inside the existing bulk column query with
    pg_get_serial_sequence(), not a second round trip — the bulk fetch has to
    stay at a constant number of queries however many tables are involved, and
    BulkSchemaPostgresTest enforces exactly that;

  - enum columns resolve through udt_name/udt_schema instead of the
    'USER-DEFINED' placeholder;

  - partitioned parents keep PARTITION BY and partitions are re-attached with
    their original bound.

Sonar findings on the PR are addressed by extracting the new behaviour into
DBDiff\DB\Support\PostgresSchemaHelper, following the pattern QueryHelper
already documents: both adapters sit on the 20-method ceiling, so shared
behaviour is extracted rather than added as another private method.

  - class method count back to 20 (was 23)
  - serialTypeFor() down from 5 returns to 2
  - the per-row column shape (identity / generated / serial / plain) moves out
    of assembleColumns() into columnDefinition(), keeping the assembly loop
    under the complexity ceiling it was already close to

Verified against PostgreSQL 14, 15, 16, 17 and 18: 46 tests / 137 assertions
each. Unit + SQLite 333 tests, unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mysql Related to Mysql php Pull requests that update php code sqlite Related to Sqlite

Projects

None yet

2 participants