test(conformance): run the 90-case renderer corpus in CI - #207
Merged
Merged
Conversation
The hard-cases corpus has been measuring this renderer for weeks — 45 of 90, then 52, then 66 — but only from a scratch script on one machine. Nothing in CI touched it, so the number was a thing I reported rather than a thing the project enforced. It now runs in the conformance harness: Tested: 238 -> 328 Passed: 220 -> 272 The 38 failures are recorded under hard_cases rather than fixed here: exclusion constraints, NULLS NOT DISTINCT, NOT VALID, foreign key variants and several partitioning forms. That count can now only go down, and the harness reports the moment one starts passing. More of them fail here than in the scratch measurement, which measured getCreateStatement alone. The harness runs the whole diff path, so this is the stricter and more honest number. `objects` is read from the package too. It was a byte-identical copy in this repository — the same duplication that let the fingerprint definitions drift, and the copy nobody looks at is the one that goes stale. The local file is deleted rather than left as a second source of truth. Verified: 328 tested and exit 0 with the baseline, and exit 1 with a NEW failure report when a baseline entry is removed, so a real regression is still caught. Co-Authored-By: Claude <noreply@anthropic.com>
PostgreSQL 18 gave NOT NULL constraints names, and LIKE ... INCLUDING ALL copies them. The renderer re-emits the *source* table's names on the new table, so the replayed schema carries src_id_not_null where the server would have derived h_id_not_null, and the fingerprints differ. Only reachable on 18 — on 16 and 17 there is no name to copy — which is why the case passed when the baseline was captured. Reproduced against a local PostgreSQL 18.6 before recording it, so this is baselined as a known renderer gap rather than as suspected CI flake. Co-Authored-By: Claude <noreply@anthropic.com>
jasdeepkhalsa
force-pushed
the
claude/hard-cases-in-ci
branch
from
September 12, 2026 11:52
4fa6c6b to
ffb098c
Compare
patterns.json is produced by extract-patterns.php, which run.sh invokes on every run before the suite reads it. The committed copy is therefore never the one CI tests against — it is a snapshot of whichever server the last local run happened to use, and extraction is live-validated, so it differs by PostgreSQL version (336 entries on one, 338 on another). Committing that snapshot put ~450 lines of churn in a change about loading the hard-case corpus, and made it read as though tests were being skipped: 26 "intentional_error" lines appeared in the diff. They were pre-existing classifications reappearing in a rewritten file — no pattern changed to intentional_error, and no classification rule was touched. Reverted to master's copy so the diff shows only the actual change. Co-Authored-By: Claude <noreply@anthropic.com>
extract-patterns.php writes this file, and run.sh runs the extractor on every invocation before the suite reads it. The committed copy was never the one CI tested against. Because extraction is live-validated against the server, its contents also depend on the PostgreSQL version that produced it — 336 entries on one version, 338 on another. So any local run rewrote the file, and those rewrites showed up as hundreds of lines of churn that read as though tests were being skipped, when no classification had changed at all. Nothing depends on it being present: the conformance job invokes run.sh, and run-conformance.php already exits with "Run extract-patterns.php first" when the file is absent. Co-Authored-By: Claude <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
The Postgres conformance harness already ran the 20-case object corpus from
@akalforge/pg-conformance. That package also ships a 90-case hard-casecorpus — partitioning, identity, generated columns, collation, exclusion
constraints,
LIKE ... INCLUDING ALL, nested composites and similar — whichnothing was running. This wires it into CI so those cases are checked on every
PR, across PostgreSQL 16, 17 and 18.
Results, plainly
Of the 90 hard cases, 51 pass and 39 are recorded in the baseline as known
failures.
Baselining is not fixing. Those 39 are genuine gaps in the renderer, and this
PR does not close them. What it does is stop them being unknown: each is named,
the suite fails if any new case breaks, and a fix can be measured by cases
moving out of the baseline rather than by assertion.
One was root-caused rather than just recorded.
hard_table_like_including_allfails only on PostgreSQL 18, which gave
NOT NULLconstraints names.LIKE ... INCLUDING ALLcopies them, and the renderer re-emits the sourcetable's constraint names on the new table, so a replayed schema carries
src_id_not_nullwhere the server would deriveh_id_not_null. Unreachable on16 and 17, where there is no name to copy — which is why it was not in the
baseline captured earlier.
Two pieces of cleanup
tests/pg-conformance/patterns-objects.jsonis deleted. It was abyte-identical copy of the corpus now loaded from the package, so it could
drift silently out of agreement with the source of truth.
tests/pg-conformance/patterns.jsonis no longer tracked. It is generatedby
extract-patterns.php, whichrun.shinvokes on every run before the suitereads it, so the committed copy was never the one CI tested against. Extraction
is live-validated against the server, so its contents depend on the PostgreSQL
version that produced it — 336 entries on one version, 338 on another. Any
local run therefore rewrote it, producing hundreds of lines of diff churn that
read as though tests were being skipped when no classification had changed.
Nothing depends on it being present:
run-conformance.phpalready exits with"Run extract-patterns.php first" when it is absent.
No skip rule was added or altered —
extract-patterns.phpis untouched by thisPR.
MySQL
Unaffected. This is the Postgres conformance harness only; no MySQL adapter,
renderer or test path is touched, and the MySQL suites are unchanged.
Verification
Conformance suite green on PostgreSQL 16, 17 and 18. The PG18 case above was
reproduced against a local PostgreSQL 18.6 server before being recorded, so it
is baselined as an understood gap rather than as a suspected CI flake.