Skip to content

fix(conformance): restore the ALTER coverage, and fail when it is missing - #202

Merged
jasdeepkhalsa merged 2 commits into
masterfrom
claude/restore-alter-coverage
Sep 3, 2026
Merged

jasdeepkhalsa merged 2 commits into
masterfrom
claude/restore-alter-coverage

Conversation

@jasdeepkhalsa

Copy link
Copy Markdown
Member

The conformance suite has been testing object creation only.

Its source for PostgreSQL's regression files — malisper/pgrust — removed its vendor/ directory. Every fetch returned 404. The script warned and carried on, the extractor wrote an empty patterns.json, and the run reported:

Loaded 20 object/create patterns
  Tested:            20
  Passed:            15

…exiting 0 and looking healthy, while the ALTER patterns the suite exists for were never exercised. That is the entire diff path for an existing schema — the thing DBDiff actually does.

This is happening on CI too, not just locally.

Restored

Pointing at PostgreSQL's own regression suite, pinned to REL_18_STABLE:

  Tested:            238      (was 20)
  Passed:            220

326 ALTER patterns are extracted again:

category patterns
add_constraint 113
add_column 75
change_type 28
drop_column 28
drop_constraint 18
set/drop_not_null 28
add/drop_identity 17
set/drop_default 19

13 failures nobody could see

category count
add_identity 7
change_type 3
drop_default 2
add_column 1

The identity cluster mirrors, on the ALTER path, the CREATE-path bug already fixed — the same blind spot, one layer along.

They are baselined under alter_path, not fixed here, so the suite is green and the debt is written down where the harness already reports it. It also reports the moment one starts passing, so they cannot be quietly forgotten.

Fails closed now

The script exits rather than degrading to a smaller suite. Verified: with an unreachable ref it exits 1 with an explanation, where it previously exited 0 having tested a fifth as much.

A suite that quietly tests less is worse than one that stops, because nothing in the output says so — which is exactly how this went unnoticed.

🤖 Generated with Claude Code

…sing

The conformance suite has been testing object creation only. Its source for
PostgreSQL's regression files — malisper/pgrust — removed its vendor/
directory, so every fetch returned 404. The script warned and carried on, the
extractor wrote an empty patterns.json, and the run reported:

  Loaded 20 object/create patterns
    Tested:            20
    Passed:            15

exiting 0 and looking healthy, while the ALTER patterns the suite exists for
were never exercised. That is the whole diff path for an existing schema.

Pointing at PostgreSQL's own regression suite, pinned to REL_18_STABLE:

    Tested:            238
    Passed:            220

326 ALTER patterns are extracted again — 113 add_constraint, 75 add_column,
28 change_type, 28 drop_column, 18 drop_constraint, plus not-null, default and
identity transitions.

That immediately exposed 13 failures nobody could see, clustered on identity:
seven add_identity, three change_type, two drop_default, one add_column. They
are baselined under alter_path rather than fixed here, so the suite is green
and the debt is written down where the harness already reports it — and it
reports the moment one starts passing.

The script now fails when the files cannot be fetched instead of degrading to
a smaller suite. Verified: with an unreachable ref it exits 1 with an
explanation, where it previously exited 0 having tested a fifth as much. A
suite that quietly tests less is worse than one that stops, because nothing
about the output says so.

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added bug postgres Related to Postgres labels Sep 3, 2026
Restoring the ALTER patterns made the suite fail on PostgreSQL 16 and 18 for a
reason that is not a regression: patterns are extracted by validating against
the live server, so the set differs by version — 235 on 16, 238 on 17, 270 on
18. A case that fails on one version can legitimately pass on another, and the
harness treated a baselined case passing as fatal. No single baseline could
ever satisfy the whole matrix.

A baselined case that passes is now reported and not fatal. A failure that is
*not* in the baseline still fails the run, because that is a real regression
and the reason the check exists.

Verified both ways on PostgreSQL 17: the suite exits 0 with one baselined case
passing, and exits 1 with a NEW failure report when a baseline entry is
removed to simulate a regression.

Also baselines constraints_add_identity_31, which only appears on PostgreSQL
18 where the extracted pattern set is larger.

Co-Authored-By: Claude <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Sep 3, 2026

Copy link
Copy Markdown

@jasdeepkhalsa
jasdeepkhalsa merged commit 83fa03e into master Sep 3, 2026
68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug postgres Related to Postgres

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant