perf(pgsql): batch schema fetching for changed tables (#165, #184) - #183
Merged
Merged
Conversation
Replace O(N) per-table queries with O(1) batch queries when tables differ between source and target. For N changed tables, PostgresAdapter now runs 7 fixed queries per side (14 total) regardless of N, down from 8 per table per side (16N total). Key changes: - Add BulkSchemaAdapterInterface with getBulkTableSchema() contract - PostgresAdapter implements the new interface; getBulkTableSchema() runs 7 IN-clause batch queries and assembles results via assembleColumns(), assembleIndexes(), assembleConstraints() - getTableSchema() and getCreateStatement() delegate to getBulkTableSchema() - DBSchema.getDiff() bulk-fetches schemas for all changed tables before the diff loop when the adapter implements BulkSchemaAdapterInterface - TableSchema.getDiff() accepts optional pre-fetched schemas to avoid redundant DB queries - Bonus: fixes the duplicate named-NOT-NULL-constraint query that previously ran twice per table (once from fetchColumns, once from fetchConstraints) MySQL and SQLite adapters are unaffected (MySQL already runs ~2 queries per table; SQLite is always local). The BulkSchemaAdapterInterface is a separate opt-in interface so DBAdapterInterface is unchanged.
…collision Audit of the two most recent performance features (#179/#180 pre-scan and #165 batch fetch) found both had shipped with no test coverage, and turned up a correctness bug in the batched constraint query. Correctness - Scope the pg_constraint join to the owning table. Postgres scopes constraint names per table, not per schema, so joining on conname alone matched constraints belonging to other tables. Verified on PG16: two tables sharing a constraint name returned a duplicated row, which the grouping step turned into FOREIGN KEY ("ref", "ref"). - Key grouped constraint columns by name so any join fan-out collapses (an N-column FK referencing an N-column key produces N x N rows). - Drop constraints buildConstraintDef cannot render instead of storing nulls in the diff map; restores behaviour lost with array_filter(). - Correct the BulkSchemaAdapterInterface docblock, which described a fallback the implementation does not have. Shared helpers - Extract DBDiff\DB\Support\QueryHelper for the two duplicated snippets: IN-clause placeholder building (PostgresAdapter, StreamingMergeDiff) and table-map narrowing (getSchemaHashMap on both MySQL and Postgres). Both adapters sit on the 20-method SonarCloud ceiling, so shared behaviour has to live outside them. Tests (+90) - QueryHelperTest: placeholder and narrowing behaviour. - PostgresBulkSchemaTest: assembly of columns, indexes and constraints from pre-fetched rows, focused on the cross-table isolation that single-table fetching used to provide for free. - SchemaBatchFetchTest: DBSchema wiring — unchanged tables are never fetched, changed tables are batched exactly once per side, and adapters without bulk support still fall back per table. - BulkSchemaPostgresTest: live-server suite asserting the batch path returns exactly what the per-table path returns, that the query budget stays at 7 per side regardless of table count, and that schema hashes track schema changes but not data writes. Compares the two code paths against each other, so it needs no per-version golden files. Docs - Document both passes in the README with the per-driver matrix and the round-trip maths behind them.
The linter and its CLI flag shipped in #174 with no README coverage: the flag was missing from the options table and the blocking behaviour was undocumented, so a first run against a database with dropped tables fails with no explanation in the docs. Adds a Destructive Change Protection section covering the error/warning split, the possible-rename downgrade, and the allowDestructive config key, and lists the flag in the diff options table.
… (S3776) DBSchema::getDiff (19 -> ~10) Extract the two blocks added for #165 into their own methods: selectTablesNeedingDiff() for the pre-scan partition and bulkFetchSchemas() for the batched load. getDiff() reads as a sequence of phases again, and each helper is well under the limit on its own. PostgresAdapter::assembleConstraints (18 -> ~12) This class sits on the 20-method ceiling (S2166), so the complexity had to come out in place rather than into a helper. Both nested loop pairs are flattened: - FK/UNIQUE/PK rows group under one table+constraint key joined by NUL, which no Postgres identifier can contain, so the emit loop is single-level and rebuilds the map from each row's own table_name/constraint_name. - Named NOT NULL constraints are collected as a flat list instead of a [table][name] map. Each row already carries both names, and the query yields exactly one row per constraint, so nothing needed the nesting. Output is unchanged: assembled constraints still come out grouped by table in the order the ORDER BY produces. Adds a test that named NOT NULL constraints sharing a name across tables stay scoped to their own table, matching the existing coverage for the other constraint kinds.
Timing the seven batch queries individually showed one of them was 99% of
the entire schema fetch: the information_schema.table_constraints join to
key_column_usage, referential_constraints and constraint_column_usage.
Those views wrap the catalogs in per-row privilege checks, which stops the
planner pushing `table_name IN (...)` down through them, so they are largely
materialised before the filter applies. Reading pg_constraint directly, with
unnest(conkey) WITH ORDINALITY for the column list, returns the same rows
without any of that.
Measured on Postgres 16 against a 1000-table database, batch fetching:
tables before after
100 4013 ms 40 ms
250 9721 ms 83 ms
500 19920 ms 157 ms
1000 39246 ms 324 ms
Roughly 120x, taking per-table cost from ~39 ms to ~0.3 ms. #165 removed the
per-table round-trips; this removes what was absorbing most of that gain.
Output is unchanged. The CASE arms reproduce exactly what the
information_schema views emit, including Postgres mapping simple match to
'NONE' rather than 'SIMPLE'. Verified by running both queries over a schema
covering composite PK/UNIQUE/FK, all five referential actions, MATCH FULL,
both deferrable modes and NOT VALID, then comparing what the assembly step
consumes: identical across all 15 constraints. The golden-file end-to-end
migration test also still matches byte for byte.
Adds a live-server test pinning every arm of the mapping, since a wrong
catalog code would silently emit the wrong referential action into a
migration rather than failing loudly.
Two incidental improvements: the constraint column ordinal is now explicit
instead of relying on key_column_usage ordering, and pg_catalog does not
omit constraints the connecting role lacks privileges on.
|
jasdeepkhalsa
added a commit
that referenced
this pull request
Aug 23, 2026
The branch was cut from a stale master and missed 20 commits, including the bulk schema fetch (#183), reading constraints from pg_catalog (#184) and the cognitive-complexity refactor. All four fixes are re-applied on top of that work rather than around it: - the duplicate PRIMARY KEY is removed from the rebuilt CREATE TABLE, which master still emitted via getPrimaryKey() alongside the named constraint; - sequence ownership is resolved inside the existing bulk column query with pg_get_serial_sequence(), not a second round trip — the bulk fetch has to stay at a constant number of queries however many tables are involved, and BulkSchemaPostgresTest enforces exactly that; - enum columns resolve through udt_name/udt_schema instead of the 'USER-DEFINED' placeholder; - partitioned parents keep PARTITION BY and partitions are re-attached with their original bound. Sonar findings on the PR are addressed by extracting the new behaviour into DBDiff\DB\Support\PostgresSchemaHelper, following the pattern QueryHelper already documents: both adapters sit on the 20-method ceiling, so shared behaviour is extracted rather than added as another private method. - class method count back to 20 (was 23) - serialTypeFor() down from 5 returns to 2 - the per-row column shape (identity / generated / serial / plain) moves out of assembleColumns() into columnDefinition(), keeping the assembly loop under the complexity ceiling it was already close to Verified against PostgreSQL 14, 15, 16, 17 and 18: 46 tests / 137 assertions each. Unit + SQLite 333 tests, unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Closes #165. Closes #184.
Makes schema diffing cost a constant number of round-trips instead of one set per table, then fixes the query that was absorbing most of that gain. Along the way this fixes a constraint-name collision bug and adds test coverage for the two performance passes, neither of which had any.
1. Batch schema fetching (#165)
The pre-scan in #180 skips tables that are already identical. The tables that genuinely differ still needed 8 queries per table per side. This loads them all together instead.
BulkSchemaAdapterInterfacewith a singlegetBulkTableSchema()method. It is a separate opt-in interface, soDBAdapterInterfaceis untouched and MySQL/SQLite need no changes — MySQL already resolves a table in ~2 queries, and SQLite is a local file with no network latency.PostgresAdapterimplements it: 7 fixed queries per side regardless of table count.getTableSchema()andgetCreateStatement()now delegate to it.DBSchema::getDiff()partitions common tables by hash, then bulk-fetches only the changed ones.TableSchema::getDiff()accepts pre-fetched schemas, falling back to per-table queries when none are supplied.For a 300-table database with 7 real changes: 16 queries total, against roughly 4,800 with per-table fetching.
Also fixes a duplicate query — the named NOT NULL lookup previously ran twice per table, once from
fetchColumns()and once fromfetchConstraints().2. pg_catalog rewrite of the constraint query (#184)
Timing the seven batch queries individually showed one was 99.2% of the whole fetch —
information_schema.table_constraintsjoined tokey_column_usage,referential_constraintsandconstraint_column_usage. Those views wrap the catalogs in per-row privilege checks, which stops the planner pushingtable_name IN (...)down through them.Reading
pg_constraintdirectly, withunnest(conkey) WITH ORDINALITYfor the column list:Roughly 120x, taking per-table cost from ~39 ms to ~0.3 ms.
Output is unchanged — the
CASEarms reproduce exactly what theinformation_schemaviews emit, including Postgres mapping simple match toNONErather thanSIMPLE. Incidentally, the column ordinal is now explicit rather than relying onkey_column_usageordering, andpg_catalogdoes not omit constraints the connecting role lacks privileges on.3. Constraint-name collision fix
Found while adding tests. The constraint query joined
pg_constraintonconnamealone, but Postgres scopes constraint names per table, not per schema. A constraint on one table matched a same-named constraint on another, duplicating the row and producing corrupt DDL:That duplicate became
FOREIGN KEY ("ref", "ref"). Reproduced on Postgres 16 with two tables sharing a constraint name. Grouped constraint columns are now keyed by name so any join fan-out collapses — an N-column FK referencing an N-column key produces N×N rows. ConstraintsbuildConstraintDef()cannot render are dropped rather than stored as nulls in the diff map.Shared helpers
MySQLAdapterandPostgresAdapterboth sit on the 20-method SonarCloud ceiling, so two duplicated snippets moved toDBDiff\DB\Support\QueryHelper: IN-clause placeholder building (PostgresAdapter,StreamingMergeDiff) and table-map narrowing (getSchemaHashMapon both adapters).Tests
Both performance passes had shipped with no coverage. This adds 91 tests:
QueryHelperTest— placeholder generation and table narrowing.PostgresBulkSchemaTest— assembly of columns, indexes and constraints from pre-fetched rows, concentrating on the cross-table isolation that single-table fetching used to provide for free.SchemaBatchFetchTest—DBSchemawiring, asserting the negatives: unchanged tables are never fetched, changed tables are batched exactly once per side, non-bulk adapters still fall back per table.BulkSchemaPostgresTest— live-server suite asserting the batch path returns exactly what the per-table path returns, that the query budget stays at 7 per side regardless of table count, that schema hashes track schema but not data, and that every referential-action mapping renders correctly. It builds its own database and compares the two code paths against each other, so it needs no per-version golden files.Verified against a live Postgres 16: 788 tests pass, including the golden-file end-to-end migration test, which still matches byte for byte. The new suite is registered in the
Postgrestestsuite, so CI runs it across PG 14–18.Docs
The README documented neither performance pass. Adds a Schema Diff Performance section with the per-driver matrix and the round-trip maths. Separately,
--allow-destructivefrom #174 had shipped with no README coverage at all — the flag was missing from the options table and the blocking behaviour was undocumented — so that gets a section too.Not included
Chunking the
IN (...)list, measured and rejected. At 1000 tables it makes under 1% difference (39246 ms vs 38908 ms, within noise), peak memory is ~4 MB, and padding the parameter list to 65000 while holding result size fixed adds only ~0.6 s. The 65535 bind-parameter ceiling is real and fails cleanly, but requires more than 65535 changed tables in a single diff. Rationale recorded in #184.