Skip to content

perf: schema pre-scan hash map to skip unchanged tables (#179 / #165) - #180

Merged
jasdeepkhalsa merged 7 commits into
masterfrom
feature/schema-prescan-parallel
Aug 5, 2026
Merged

jasdeepkhalsa merged 7 commits into
masterfrom
feature/schema-prescan-parallel

Conversation

@jasdeepkhalsa

@jasdeepkhalsa jasdeepkhalsa commented Aug 4, 2026

Copy link
Copy Markdown
Member

Problem

When diffing a large PostgreSQL / Supabase database, DBSchema::getDiff() fires 7–14 sequential queries per common table (3 in fetchColumns, 2 in fetchIndexes, 2 in fetchConstraints). Over a high-latency connection with 100+ common tables this adds up to 1,000–1,400 round trips and routinely hits the Supabase timeout. Resolves #179 and #165.

Solution — schema pre-scan hash map

Before entering the per-table loop, fire two batch queries (one per DB side) that return a schema hash for every table in a single round trip. Tables whose hashes match are identical and are skipped entirely.

Before (100 common tables, 5 changed):  ~1,400 queries
After  (100 common tables, 5 changed):  ~16 queries
                                         (2 pre-scan + ~14 for 5 changed tables)

The pre-scan is zero-risk: if a hash is missing for any reason the table falls through to the normal full diff.

Changes

File What changed
src/DB/Adapters/DBAdapterInterface.php New getSchemaHashMap(Connection, array): array contract
src/DB/Adapters/PostgresAdapter.php Single CTE query joining information_schema.columns, pg_indexes, information_schema.table_constraints (with FK deferral/rules), and pg_constraint (CHECK / EXCLUDE / named NOT NULL). Returns md5(...) per table.
src/DB/Adapters/MySQLAdapter.php Four queries total: SHOW TABLE STATUS + INFORMATION_SCHEMA.COLUMNS + INFORMATION_SCHEMA.STATISTICS + KEY_COLUMN_USAGE ⋈ REFERENTIAL_CONSTRAINTS. Combined in PHP into one md5() per table.
src/DB/Adapters/SQLiteAdapter.php Returns [] — SQLite is a local file, no latency overhead; falls back to per-table diffs.
src/DB/DBManager.php Thin getSchemaHashMap(string, array) wrapper
src/DB/Schema/DBSchema.php Pre-scan + skip loop before $commonTables iteration; logs count of skipped tables via Logger::info

Hash coverage

The hash is designed to change whenever getTableSchema() would return something different:

  • Postgres: column names/types/defaults/nullability/identity/generated expressions + all index definitions + all constraint names/types/FK rules/deferral + CHECK/EXCLUDE definitions + named NOT NULL constraints (PG18+)
  • MySQL: engine/collation + column names/types/defaults/nullability/extra + index name/column/uniqueness + FK constraint names/columns/rules

Fallback safety

if (isset($sourceHashes[$table], $targetHashes[$table])
    && $sourceHashes[$table] === $targetHashes[$table]) {
    $skipped++;
    continue;  // both sides returned a hash AND they match
}
// otherwise: full per-table diff as before

A missing hash (adapter returns [], table absent from result, any query failure) means the table is always diffed — no silent skips.


Generated by Claude Code

Adds a two-query pre-scan phase before the per-table diff loop in
DBSchema::getDiff(). For each DB connection a single batch query
returns an MD5 hash covering columns, indexes, and constraints for
every common table. Tables whose hashes match are skipped entirely,
eliminating 7-14 sequential queries per unchanged table.

For a Supabase instance with 100 common tables where 5 have changed,
query count drops from ~1400 to ~16 (2 pre-scan + ~14 for 5 changed
tables). This resolves the timeout seen when diffing large self-hosted
Supabase databases.

• DBAdapterInterface: new getSchemaHashMap(Connection, array): array
• MySQLAdapter: 4-query implementation (SHOW TABLE STATUS + 3 INFO_SCHEMA)
• PostgresAdapter: single CTE query covering columns/indexes/constraints
• SQLiteAdapter: returns [] → all tables diffed normally (local file, no latency)
• DBManager: thin wrapper getSchemaHashMap(string, array)
• DBSchema: pre-scan before commonTables loop with Logger info line
@github-actions github-actions Bot added mysql Related to Mysql php Pull requests that update php code postgres Related to Postgres sqlite Related to Sqlite labels Aug 4, 2026
claude added 6 commits August 4, 2026 23:57
…can CTE

PostgreSQL forbids null characters (U+0000) in text strings. Using chr(0) as
a COALESCE sentinel inside string_agg caused SQLSTATE 54000 on any database
whose column_default values included NULL rows. Replace with an empty string,
which is safe and sufficient for distinguishing NULL vs empty defaults in the
hash comparison context.

Co-Authored-By: Claude <noreply@anthropic.com>
… STATUS

SHOW TABLE STATUS FROM `$db` interpolated the database name directly into
SQL, triggering a SonarCloud security hotspot (SQL injection risk). Replace
with a parameterized INFORMATION_SCHEMA.TABLES query that is functionally
equivalent and uses a bound parameter for the schema name.

Co-Authored-By: Claude <noreply@anthropic.com>
SonarCloud rule php:S4790 flags md5() as a weak hashing algorithm
regardless of context. The hash is used only for schema fingerprinting
(equality comparison), not cryptographic security, but sha256 satisfies
the rule and produces equally reliable fingerprints for this use case.

Co-Authored-By: Claude <noreply@anthropic.com>
…776)

SonarCloud S3776 flags functions with cognitive complexity > 15.
getSchemaHashMap() accumulated ~26 points from 5 loops, nested conditionals,
ternaries, and null-coalescing operators. Extract each of the four query+loop
blocks into private fetchEngineMap/ColMap/IdxMap/ConMap helpers, bringing
the public method down to ~8 and each helper to 1-2.

Co-Authored-By: Claude <noreply@anthropic.com>
… class)

MySQLAdapter and PostgresAdapter each reached 21 methods after adding
getSchemaHashMap(). Fix by:
- MySQLAdapter: inline fetchEngineMap() back into getSchemaHashMap() (trivial
  one-loop body, not worth a separate method)
- PostgresAdapter: merge resolveParameterisedType() into buildColumnType()
  using lookup arrays for varchar/char and timestamptz to keep cognitive
  complexity at 14 (threshold: 15)

Both classes now have exactly 20 declared methods.

Co-Authored-By: Claude <noreply@anthropic.com>
Restructure buildColumnType() so it has exactly 3 return statements
(early exit for domain_name, early exit for simple-map lookup, and a
single final return $result). The if/elseif chain sets $result in-place
rather than returning early, keeping cognitive complexity at 14.

Co-Authored-By: Claude <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Aug 5, 2026

Copy link
Copy Markdown

@jasdeepkhalsa
jasdeepkhalsa merged commit d4165a8 into master Aug 5, 2026
67 checks passed
jasdeepkhalsa pushed a commit that referenced this pull request Aug 10, 2026
…collision

Audit of the two most recent performance features (#179/#180 pre-scan and
#165 batch fetch) found both had shipped with no test coverage, and turned
up a correctness bug in the batched constraint query.

Correctness
- Scope the pg_constraint join to the owning table. Postgres scopes
  constraint names per table, not per schema, so joining on conname alone
  matched constraints belonging to other tables. Verified on PG16: two
  tables sharing a constraint name returned a duplicated row, which the
  grouping step turned into FOREIGN KEY ("ref", "ref").
- Key grouped constraint columns by name so any join fan-out collapses
  (an N-column FK referencing an N-column key produces N x N rows).
- Drop constraints buildConstraintDef cannot render instead of storing
  nulls in the diff map; restores behaviour lost with array_filter().
- Correct the BulkSchemaAdapterInterface docblock, which described a
  fallback the implementation does not have.

Shared helpers
- Extract DBDiff\DB\Support\QueryHelper for the two duplicated snippets:
  IN-clause placeholder building (PostgresAdapter, StreamingMergeDiff) and
  table-map narrowing (getSchemaHashMap on both MySQL and Postgres). Both
  adapters sit on the 20-method SonarCloud ceiling, so shared behaviour has
  to live outside them.

Tests (+90)
- QueryHelperTest: placeholder and narrowing behaviour.
- PostgresBulkSchemaTest: assembly of columns, indexes and constraints from
  pre-fetched rows, focused on the cross-table isolation that single-table
  fetching used to provide for free.
- SchemaBatchFetchTest: DBSchema wiring — unchanged tables are never
  fetched, changed tables are batched exactly once per side, and adapters
  without bulk support still fall back per table.
- BulkSchemaPostgresTest: live-server suite asserting the batch path
  returns exactly what the per-table path returns, that the query budget
  stays at 7 per side regardless of table count, and that schema hashes
  track schema changes but not data writes. Compares the two code paths
  against each other, so it needs no per-version golden files.

Docs
- Document both passes in the README with the per-driver matrix and the
  round-trip maths behind them.
jasdeepkhalsa added a commit that referenced this pull request Aug 10, 2026
)

Closes #165. Closes #184.

Makes schema diffing cost a constant number of round-trips instead of
one set per table, then fixes the query that was absorbing most of that
gain. Along the way this fixes a constraint-name collision bug and adds
test coverage for the two performance passes, neither of which had any.

## 1. Batch schema fetching (#165)

The pre-scan in #180 skips tables that are already identical. The tables
that genuinely differ still needed 8 queries per table per side. This
loads them all together instead.

- New `BulkSchemaAdapterInterface` with a single `getBulkTableSchema()`
method. It is a separate opt-in interface, so `DBAdapterInterface` is
untouched and MySQL/SQLite need no changes — MySQL already resolves a
table in ~2 queries, and SQLite is a local file with no network latency.
- `PostgresAdapter` implements it: 7 fixed queries per side regardless
of table count. `getTableSchema()` and `getCreateStatement()` now
delegate to it.
- `DBSchema::getDiff()` partitions common tables by hash, then
bulk-fetches only the changed ones.
- `TableSchema::getDiff()` accepts pre-fetched schemas, falling back to
per-table queries when none are supplied.

For a 300-table database with 7 real changes: 16 queries total, against
roughly 4,800 with per-table fetching.

Also fixes a duplicate query — the named NOT NULL lookup previously ran
twice per table, once from `fetchColumns()` and once from
`fetchConstraints()`.

## 2. pg_catalog rewrite of the constraint query (#184)

Timing the seven batch queries individually showed one was **99.2%** of
the whole fetch — `information_schema.table_constraints` joined to
`key_column_usage`, `referential_constraints` and
`constraint_column_usage`. Those views wrap the catalogs in per-row
privilege checks, which stops the planner pushing `table_name IN (...)`
down through them.

Reading `pg_constraint` directly, with `unnest(conkey) WITH ORDINALITY`
for the column list:

| tables in batch | before | after |
| ---: | ---: | ---: |
| 100 | 4013 ms | 40 ms |
| 250 | 9721 ms | 83 ms |
| 500 | 19920 ms | 157 ms |
| 1000 | 39246 ms | 324 ms |

Roughly 120x, taking per-table cost from ~39 ms to ~0.3 ms.

Output is unchanged — the `CASE` arms reproduce exactly what the
`information_schema` views emit, including Postgres mapping simple match
to `NONE` rather than `SIMPLE`. Incidentally, the column ordinal is now
explicit rather than relying on `key_column_usage` ordering, and
`pg_catalog` does not omit constraints the connecting role lacks
privileges on.

## 3. Constraint-name collision fix

Found while adding tests. The constraint query joined `pg_constraint` on
`conname` alone, but **Postgres scopes constraint names per table, not
per schema**. A constraint on one table matched a same-named constraint
on another, duplicating the row and producing corrupt DDL:

```
old (name-only join):  users | shared_name | ref     <- duplicated
                       users | shared_name | ref
new (table-scoped):    users | shared_name | ref
```

That duplicate became `FOREIGN KEY ("ref", "ref")`. Reproduced on
Postgres 16 with two tables sharing a constraint name. Grouped
constraint columns are now keyed by name so any join fan-out collapses —
an N-column FK referencing an N-column key produces N×N rows.
Constraints `buildConstraintDef()` cannot render are dropped rather than
stored as nulls in the diff map.

## Shared helpers

`MySQLAdapter` and `PostgresAdapter` both sit on the 20-method
SonarCloud ceiling, so two duplicated snippets moved to
`DBDiff\DB\Support\QueryHelper`: IN-clause placeholder building
(`PostgresAdapter`, `StreamingMergeDiff`) and table-map narrowing
(`getSchemaHashMap` on both adapters).

## Tests

Both performance passes had shipped with no coverage. This adds 91
tests:

- **`QueryHelperTest`** — placeholder generation and table narrowing.
- **`PostgresBulkSchemaTest`** — assembly of columns, indexes and
constraints from pre-fetched rows, concentrating on the cross-table
isolation that single-table fetching used to provide for free.
- **`SchemaBatchFetchTest`** — `DBSchema` wiring, asserting the
negatives: unchanged tables are never fetched, changed tables are
batched exactly once per side, non-bulk adapters still fall back per
table.
- **`BulkSchemaPostgresTest`** — live-server suite asserting the batch
path returns exactly what the per-table path returns, that the query
budget stays at 7 per side regardless of table count, that schema hashes
track schema but not data, and that every referential-action mapping
renders correctly. It builds its own database and compares the two code
paths against each other, so it needs no per-version golden files.

Verified against a live Postgres 16: 788 tests pass, including the
golden-file end-to-end migration test, which still matches byte for
byte. The new suite is registered in the `Postgres` testsuite, so CI
runs it across PG 14–18.

## Docs

The README documented neither performance pass. Adds a **Schema Diff
Performance** section with the per-driver matrix and the round-trip
maths. Separately, `--allow-destructive` from #174 had shipped with no
README coverage at all — the flag was missing from the options table and
the blocking behaviour was undocumented — so that gets a section too.

## Not included

Chunking the `IN (...)` list, measured and rejected. At 1000 tables it
makes under 1% difference (39246 ms vs 38908 ms, within noise), peak
memory is ~4 MB, and padding the parameter list to 65000 while holding
result size fixed adds only ~0.6 s. The 65535 bind-parameter ceiling is
real and fails cleanly, but requires more than 65535 *changed* tables in
a single diff. Rationale recorded in #184.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mysql Related to Mysql php Pull requests that update php code postgres Related to Postgres sqlite Related to Sqlite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature: schema-level checksum pre-scan to skip unchanged tables

2 participants