fix(conformance): restore the ALTER coverage, and fail when it is missing - #202
Merged
Merged
Conversation
…sing
The conformance suite has been testing object creation only. Its source for
PostgreSQL's regression files — malisper/pgrust — removed its vendor/
directory, so every fetch returned 404. The script warned and carried on, the
extractor wrote an empty patterns.json, and the run reported:
Loaded 20 object/create patterns
Tested: 20
Passed: 15
exiting 0 and looking healthy, while the ALTER patterns the suite exists for
were never exercised. That is the whole diff path for an existing schema.
Pointing at PostgreSQL's own regression suite, pinned to REL_18_STABLE:
Tested: 238
Passed: 220
326 ALTER patterns are extracted again — 113 add_constraint, 75 add_column,
28 change_type, 28 drop_column, 18 drop_constraint, plus not-null, default and
identity transitions.
That immediately exposed 13 failures nobody could see, clustered on identity:
seven add_identity, three change_type, two drop_default, one add_column. They
are baselined under alter_path rather than fixed here, so the suite is green
and the debt is written down where the harness already reports it — and it
reports the moment one starts passing.
The script now fails when the files cannot be fetched instead of degrading to
a smaller suite. Verified: with an unreachable ref it exits 1 with an
explanation, where it previously exited 0 having tested a fifth as much. A
suite that quietly tests less is worse than one that stops, because nothing
about the output says so.
Co-Authored-By: Claude <noreply@anthropic.com>
Restoring the ALTER patterns made the suite fail on PostgreSQL 16 and 18 for a reason that is not a regression: patterns are extracted by validating against the live server, so the set differs by version — 235 on 16, 238 on 17, 270 on 18. A case that fails on one version can legitimately pass on another, and the harness treated a baselined case passing as fatal. No single baseline could ever satisfy the whole matrix. A baselined case that passes is now reported and not fatal. A failure that is *not* in the baseline still fails the run, because that is a real regression and the reason the check exists. Verified both ways on PostgreSQL 17: the suite exits 0 with one baselined case passing, and exits 1 with a NEW failure report when a baseline entry is removed to simulate a regression. Also baselines constraints_add_identity_31, which only appears on PostgreSQL 18 where the extracted pattern set is larger. Co-Authored-By: Claude <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



The conformance suite has been testing object creation only.
Its source for PostgreSQL's regression files —
malisper/pgrust— removed itsvendor/directory. Every fetch returned 404. The script warned and carried on, the extractor wrote an emptypatterns.json, and the run reported:…exiting 0 and looking healthy, while the ALTER patterns the suite exists for were never exercised. That is the entire diff path for an existing schema — the thing DBDiff actually does.
This is happening on CI too, not just locally.
Restored
Pointing at PostgreSQL's own regression suite, pinned to
REL_18_STABLE:326 ALTER patterns are extracted again:
13 failures nobody could see
The identity cluster mirrors, on the ALTER path, the CREATE-path bug already fixed — the same blind spot, one layer along.
They are baselined under
alter_path, not fixed here, so the suite is green and the debt is written down where the harness already reports it. It also reports the moment one starts passing, so they cannot be quietly forgotten.Fails closed now
The script exits rather than degrading to a smaller suite. Verified: with an unreachable ref it exits 1 with an explanation, where it previously exited 0 having tested a fifth as much.
A suite that quietly tests less is worse than one that stops, because nothing in the output says so — which is exactly how this went unnoticed.
🤖 Generated with Claude Code