perf: schema pre-scan hash map to skip unchanged tables (#179 / #165) - #180
Merged
Merged
Conversation
Adds a two-query pre-scan phase before the per-table diff loop in DBSchema::getDiff(). For each DB connection a single batch query returns an MD5 hash covering columns, indexes, and constraints for every common table. Tables whose hashes match are skipped entirely, eliminating 7-14 sequential queries per unchanged table. For a Supabase instance with 100 common tables where 5 have changed, query count drops from ~1400 to ~16 (2 pre-scan + ~14 for 5 changed tables). This resolves the timeout seen when diffing large self-hosted Supabase databases. • DBAdapterInterface: new getSchemaHashMap(Connection, array): array • MySQLAdapter: 4-query implementation (SHOW TABLE STATUS + 3 INFO_SCHEMA) • PostgresAdapter: single CTE query covering columns/indexes/constraints • SQLiteAdapter: returns [] → all tables diffed normally (local file, no latency) • DBManager: thin wrapper getSchemaHashMap(string, array) • DBSchema: pre-scan before commonTables loop with Logger info line
…can CTE PostgreSQL forbids null characters (U+0000) in text strings. Using chr(0) as a COALESCE sentinel inside string_agg caused SQLSTATE 54000 on any database whose column_default values included NULL rows. Replace with an empty string, which is safe and sufficient for distinguishing NULL vs empty defaults in the hash comparison context. Co-Authored-By: Claude <noreply@anthropic.com>
… STATUS SHOW TABLE STATUS FROM `$db` interpolated the database name directly into SQL, triggering a SonarCloud security hotspot (SQL injection risk). Replace with a parameterized INFORMATION_SCHEMA.TABLES query that is functionally equivalent and uses a bound parameter for the schema name. Co-Authored-By: Claude <noreply@anthropic.com>
SonarCloud rule php:S4790 flags md5() as a weak hashing algorithm regardless of context. The hash is used only for schema fingerprinting (equality comparison), not cryptographic security, but sha256 satisfies the rule and produces equally reliable fingerprints for this use case. Co-Authored-By: Claude <noreply@anthropic.com>
…776) SonarCloud S3776 flags functions with cognitive complexity > 15. getSchemaHashMap() accumulated ~26 points from 5 loops, nested conditionals, ternaries, and null-coalescing operators. Extract each of the four query+loop blocks into private fetchEngineMap/ColMap/IdxMap/ConMap helpers, bringing the public method down to ~8 and each helper to 1-2. Co-Authored-By: Claude <noreply@anthropic.com>
… class) MySQLAdapter and PostgresAdapter each reached 21 methods after adding getSchemaHashMap(). Fix by: - MySQLAdapter: inline fetchEngineMap() back into getSchemaHashMap() (trivial one-loop body, not worth a separate method) - PostgresAdapter: merge resolveParameterisedType() into buildColumnType() using lookup arrays for varchar/char and timestamptz to keep cognitive complexity at 14 (threshold: 15) Both classes now have exactly 20 declared methods. Co-Authored-By: Claude <noreply@anthropic.com>
Restructure buildColumnType() so it has exactly 3 return statements (early exit for domain_name, early exit for simple-map lookup, and a single final return $result). The if/elseif chain sets $result in-place rather than returning early, keeping cognitive complexity at 14. Co-Authored-By: Claude <noreply@anthropic.com>
|
jasdeepkhalsa
pushed a commit
that referenced
this pull request
Aug 10, 2026
…collision Audit of the two most recent performance features (#179/#180 pre-scan and #165 batch fetch) found both had shipped with no test coverage, and turned up a correctness bug in the batched constraint query. Correctness - Scope the pg_constraint join to the owning table. Postgres scopes constraint names per table, not per schema, so joining on conname alone matched constraints belonging to other tables. Verified on PG16: two tables sharing a constraint name returned a duplicated row, which the grouping step turned into FOREIGN KEY ("ref", "ref"). - Key grouped constraint columns by name so any join fan-out collapses (an N-column FK referencing an N-column key produces N x N rows). - Drop constraints buildConstraintDef cannot render instead of storing nulls in the diff map; restores behaviour lost with array_filter(). - Correct the BulkSchemaAdapterInterface docblock, which described a fallback the implementation does not have. Shared helpers - Extract DBDiff\DB\Support\QueryHelper for the two duplicated snippets: IN-clause placeholder building (PostgresAdapter, StreamingMergeDiff) and table-map narrowing (getSchemaHashMap on both MySQL and Postgres). Both adapters sit on the 20-method SonarCloud ceiling, so shared behaviour has to live outside them. Tests (+90) - QueryHelperTest: placeholder and narrowing behaviour. - PostgresBulkSchemaTest: assembly of columns, indexes and constraints from pre-fetched rows, focused on the cross-table isolation that single-table fetching used to provide for free. - SchemaBatchFetchTest: DBSchema wiring — unchanged tables are never fetched, changed tables are batched exactly once per side, and adapters without bulk support still fall back per table. - BulkSchemaPostgresTest: live-server suite asserting the batch path returns exactly what the per-table path returns, that the query budget stays at 7 per side regardless of table count, and that schema hashes track schema changes but not data writes. Compares the two code paths against each other, so it needs no per-version golden files. Docs - Document both passes in the README with the per-driver matrix and the round-trip maths behind them.
jasdeepkhalsa
added a commit
that referenced
this pull request
Aug 10, 2026
) Closes #165. Closes #184. Makes schema diffing cost a constant number of round-trips instead of one set per table, then fixes the query that was absorbing most of that gain. Along the way this fixes a constraint-name collision bug and adds test coverage for the two performance passes, neither of which had any. ## 1. Batch schema fetching (#165) The pre-scan in #180 skips tables that are already identical. The tables that genuinely differ still needed 8 queries per table per side. This loads them all together instead. - New `BulkSchemaAdapterInterface` with a single `getBulkTableSchema()` method. It is a separate opt-in interface, so `DBAdapterInterface` is untouched and MySQL/SQLite need no changes — MySQL already resolves a table in ~2 queries, and SQLite is a local file with no network latency. - `PostgresAdapter` implements it: 7 fixed queries per side regardless of table count. `getTableSchema()` and `getCreateStatement()` now delegate to it. - `DBSchema::getDiff()` partitions common tables by hash, then bulk-fetches only the changed ones. - `TableSchema::getDiff()` accepts pre-fetched schemas, falling back to per-table queries when none are supplied. For a 300-table database with 7 real changes: 16 queries total, against roughly 4,800 with per-table fetching. Also fixes a duplicate query — the named NOT NULL lookup previously ran twice per table, once from `fetchColumns()` and once from `fetchConstraints()`. ## 2. pg_catalog rewrite of the constraint query (#184) Timing the seven batch queries individually showed one was **99.2%** of the whole fetch — `information_schema.table_constraints` joined to `key_column_usage`, `referential_constraints` and `constraint_column_usage`. Those views wrap the catalogs in per-row privilege checks, which stops the planner pushing `table_name IN (...)` down through them. Reading `pg_constraint` directly, with `unnest(conkey) WITH ORDINALITY` for the column list: | tables in batch | before | after | | ---: | ---: | ---: | | 100 | 4013 ms | 40 ms | | 250 | 9721 ms | 83 ms | | 500 | 19920 ms | 157 ms | | 1000 | 39246 ms | 324 ms | Roughly 120x, taking per-table cost from ~39 ms to ~0.3 ms. Output is unchanged — the `CASE` arms reproduce exactly what the `information_schema` views emit, including Postgres mapping simple match to `NONE` rather than `SIMPLE`. Incidentally, the column ordinal is now explicit rather than relying on `key_column_usage` ordering, and `pg_catalog` does not omit constraints the connecting role lacks privileges on. ## 3. Constraint-name collision fix Found while adding tests. The constraint query joined `pg_constraint` on `conname` alone, but **Postgres scopes constraint names per table, not per schema**. A constraint on one table matched a same-named constraint on another, duplicating the row and producing corrupt DDL: ``` old (name-only join): users | shared_name | ref <- duplicated users | shared_name | ref new (table-scoped): users | shared_name | ref ``` That duplicate became `FOREIGN KEY ("ref", "ref")`. Reproduced on Postgres 16 with two tables sharing a constraint name. Grouped constraint columns are now keyed by name so any join fan-out collapses — an N-column FK referencing an N-column key produces N×N rows. Constraints `buildConstraintDef()` cannot render are dropped rather than stored as nulls in the diff map. ## Shared helpers `MySQLAdapter` and `PostgresAdapter` both sit on the 20-method SonarCloud ceiling, so two duplicated snippets moved to `DBDiff\DB\Support\QueryHelper`: IN-clause placeholder building (`PostgresAdapter`, `StreamingMergeDiff`) and table-map narrowing (`getSchemaHashMap` on both adapters). ## Tests Both performance passes had shipped with no coverage. This adds 91 tests: - **`QueryHelperTest`** — placeholder generation and table narrowing. - **`PostgresBulkSchemaTest`** — assembly of columns, indexes and constraints from pre-fetched rows, concentrating on the cross-table isolation that single-table fetching used to provide for free. - **`SchemaBatchFetchTest`** — `DBSchema` wiring, asserting the negatives: unchanged tables are never fetched, changed tables are batched exactly once per side, non-bulk adapters still fall back per table. - **`BulkSchemaPostgresTest`** — live-server suite asserting the batch path returns exactly what the per-table path returns, that the query budget stays at 7 per side regardless of table count, that schema hashes track schema but not data, and that every referential-action mapping renders correctly. It builds its own database and compares the two code paths against each other, so it needs no per-version golden files. Verified against a live Postgres 16: 788 tests pass, including the golden-file end-to-end migration test, which still matches byte for byte. The new suite is registered in the `Postgres` testsuite, so CI runs it across PG 14–18. ## Docs The README documented neither performance pass. Adds a **Schema Diff Performance** section with the per-driver matrix and the round-trip maths. Separately, `--allow-destructive` from #174 had shipped with no README coverage at all — the flag was missing from the options table and the blocking behaviour was undocumented — so that gets a section too. ## Not included Chunking the `IN (...)` list, measured and rejected. At 1000 tables it makes under 1% difference (39246 ms vs 38908 ms, within noise), peak memory is ~4 MB, and padding the parameter list to 65000 while holding result size fixed adds only ~0.6 s. The 65535 bind-parameter ceiling is real and fails cleanly, but requires more than 65535 *changed* tables in a single diff. Rationale recorded in #184.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Problem
When diffing a large PostgreSQL / Supabase database,
DBSchema::getDiff()fires 7–14 sequential queries per common table (3 infetchColumns, 2 infetchIndexes, 2 infetchConstraints). Over a high-latency connection with 100+ common tables this adds up to 1,000–1,400 round trips and routinely hits the Supabase timeout. Resolves #179 and #165.Solution — schema pre-scan hash map
Before entering the per-table loop, fire two batch queries (one per DB side) that return a schema hash for every table in a single round trip. Tables whose hashes match are identical and are skipped entirely.
The pre-scan is zero-risk: if a hash is missing for any reason the table falls through to the normal full diff.
Changes
src/DB/Adapters/DBAdapterInterface.phpgetSchemaHashMap(Connection, array): arraycontractsrc/DB/Adapters/PostgresAdapter.phpinformation_schema.columns,pg_indexes,information_schema.table_constraints(with FK deferral/rules), andpg_constraint(CHECK / EXCLUDE / named NOT NULL). Returnsmd5(...)per table.src/DB/Adapters/MySQLAdapter.phpSHOW TABLE STATUS+INFORMATION_SCHEMA.COLUMNS+INFORMATION_SCHEMA.STATISTICS+KEY_COLUMN_USAGE ⋈ REFERENTIAL_CONSTRAINTS. Combined in PHP into onemd5()per table.src/DB/Adapters/SQLiteAdapter.php[]— SQLite is a local file, no latency overhead; falls back to per-table diffs.src/DB/DBManager.phpgetSchemaHashMap(string, array)wrappersrc/DB/Schema/DBSchema.php$commonTablesiteration; logs count of skipped tables viaLogger::infoHash coverage
The hash is designed to change whenever
getTableSchema()would return something different:Fallback safety
A missing hash (adapter returns
[], table absent from result, any query failure) means the table is always diffed — no silent skips.Generated by Claude Code