ci(release): complete the candidate-aware promotion pipeline on release-1.6 - #3893
Merged
Merged
Conversation
…se-1.6
Promote RC refuses to promote a release whose target base cannot verify
the packages candidate. It checks five markers on the base branch before
its first registry write and fails closed with "Target base
'release-1.6' lacks '<path>'". This branch satisfied one of the five
(hack/lib/image-refs.sh), so the next patch release off release-1.6
could not be dispatched at all.
Add the two halves the base has to carry:
* hack/verify-promoted-packages.sh and hack/lib/promoted-packages.sh,
byte-identical to main. The verifier pulls the candidate by its
committed digest and proves it matches the merge tree apart from the
artifact's impossible self-reference, that promotion rewrote every
version-line image tag, and that the container repo@digest set is
unchanged from the rc artifact.
* the two jobs that run it. verify-release-candidate in
pull-requests.yaml checks GitHub's prospective merge tree so a
packages/ edit pushed to the promote PR shows up as a failed check,
and a Verify stable packages candidate step in the finalize job of
pull-requests-release.yaml repeats it against the real merge commit.
Position is the whole point of the finalize half: it runs before the
write-once stable tag, the draft-release publish and the retag, so a bad
candidate is refused while nothing stable-named exists yet. The former
"Set up toolchain (skopeo, yq, helm)" and "Login to registry (GHCR)"
steps moved up with it and are no longer duplicated later in the job;
the toolchain now also installs flux and pins yq by version and sha256,
because an irreversible step must not depend on whatever
releases/latest/download resolves to today.
verify-release-candidate is keyed on the promote PR's author and its
head-branch shape rather than on the `release` label main matches. This
branch's on.pull_request.types has no `labeled` event, and the label is
applied about a second after the PR opens (a separate call in
gh pr create), so the `opened` payload carries no labels and a
label-keyed guard would never fire here.
.release-tooling is gitignored, matching main, so the trusted-base
checkout can never be staged.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Workflows run from the ref they fire on, so a tag pushed on this branch
executes THIS tags.yaml, not main's. main grew an rc-e2e job on
2026-07-24; release-1.6 never did. The v1.6.1-rc.1 push therefore
produced only Prepare Release, a skipped changelog and skipped website
docs — no E2E Release Candidate job at all. Promote RC looks for its
evidence by exact job name, found none, and v1.6.1 was promoted with the
gate bypassed. That is the hole this closes.
Add e2e-tag.yaml byte-identical to main and call it from a new rc-e2e
job. The suite step works as written here because this branch already
carries hack/e2e-chainsaw/ and a test-chainsaw target, and the sandbox
image already installs chainsaw 0.2.15.
Two things make the wiring load-bearing rather than cosmetic:
* the called job's name must stay "E2E ${{ inputs.tag }} (full suite)".
Promote RC composes the string it searches for from the rc tag, and
GitHub renders a called job as "<caller> / <called>", which the
gate's suffix matcher accepts. Rename either side and the gate fails
closed.
* the caller must grant contents: read AND checks: write. A caller's
permissions are the ceiling for every job in the called workflow and
GitHub validates that ceiling statically, at run creation, before any
`if:` is evaluated. e2e-tag.yaml's e2e job declares checks: write for
a dispatch-only breakpoint workflow_call can never reach, so omitting
the grant would fail the entire tags.yaml run for EVERY tag push, rc
and stable alike, building nothing.
prepare-release already publishes the rc as a non-draft prerelease with
the nocloud-amd64.raw.xz asset and pushes the release-X.Y.Z-rc.N staging
branch before rc-e2e can start, which is exactly what e2e-tag.yaml's
resolve job requires.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Since the promote PR is based on release-1.6, the changelog it carries merges onto the maintenance line and never reaches main. The backstop job in tags.yaml only asks whether the file is on origin/main, so it sees "absent", spends an AI run, and produces text nobody reviewed — which update-releasenotes.yaml then pushes into the published release body, overwriting the reviewed notes the release actually shipped with. Ask the second question too. `at_tag` is whether the changelog is present in this checkout, which is the tag commit and therefore the promote merge commit. When it is, port that reviewed file to main verbatim and skip generation entirely; the AI now runs only when the changelog is genuinely absent on both sides. Add hack/validate-changelog.sh, byte-identical to main, and gate the commit on it. A ported file came from the tag and a generated one came from a step that is deliberately allowed to fail, so neither had been checked: the old guard only asked whether the file was non-empty, which a truncated Copilot stream satisfies. The validator asserts the H1, the release-link comment, the trailing compare link and at least one section, all carrying the expected version, so a fragment or a changelog generated against the wrong tag cannot reach a release body. Verified against this branch's own shipped changelogs: v1.6.1, v1.5.2 and v1.5.1 all pass, and v1.6.1 checked against 1.6.2 fails as it should. This takes effect for tags cut after it lands — tags.yaml and the script it calls both come from the tag's own tree. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/verify-promoted-packages_test.bats is byte-identical to main: 13
tests driving the verifier against a mocked `flux`, so they need no
registry. They cover the fail-open shapes that matter most — an
unreadable file during the rc-reference scan, a collector that returns
nothing, a digest that moved while candidate and merge tree still match.
hack/promote-gate-contract.bats is NOT main's copy, and the difference is
deliberate. main's version pins a `parse` job, a `skip_e2e_gate` input,
promote-time changelog and website-docs jobs, and the `labeled`
pull_request trigger they depend on — none of which exist here, because
`Promote RC` is a workflow_dispatch and runs from the ref it is
dispatched from, which is main. Run verbatim against this branch it fails
on its first test and the runner aborts the suite.
So this pins the half release-1.6 actually owns — the producer side:
* an rc tag push calls e2e-tag.yaml with the pushed ref, only for
-rc. tags, and grants the static permission ceiling the called
workflow declares;
* the called job's name is the exact literal main's gate composes and
searches for, asserted with grep -F because `${{` as a basic regular
expression matches nothing and would make the pin pass by finding its
own subject absent;
* the suite it runs is the full one (empty CHAINSAW_SUITES), with the
Makefile target and the sandbox chainsaw binary it needs;
* all five files promote-rc.yaml's preflight reads off the target base
are present, plus the two libraries the verifier sources by relative
path;
* verify-release-candidate is keyed on author and head branch and
carries NO label condition, so restoring main's from muscle memory
switches the job off here and fails this test instead;
* the candidate verification precedes the stable tag, the submodule
tag, the release publish, the retag and the chart publish, and
follows the toolchain, the login and the tooling checkout it needs;
* the tag-time changelog is validated and ported, not regenerated.
Every pin was mutation-checked: breaking the thing it describes turns it
red.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v1.6.1 was promoted with the rc e2e gate bypassed, and its promote PR body says so verbatim. The reason is mechanical rather than anyone's judgement call: workflows run from the ref they fire on, release-1.6's
tags.yamlhas norc-e2ejob and this branch has noe2e-tag.yamlat all, so whenv1.6.1-rc.1was pushed there was nothing to produce evidence the gate could find. Bypass was the only outcome available.This closes that. Adds
e2e-tag.yamlverbatim from main, which works as written here becausehack/e2e-chainsaw/and thetest-chainsawtarget already exist on this branch. Splices main'src-e2ejob intotags.yaml. Adds theverify-release-candidatejob topull-requests.yaml, and the candidate-verification step topull-requests-release.yamlpositioned before tag creation, publish and retag. Plushack/verify-promoted-packages.sh,hack/lib/promoted-packages.sh,hack/validate-changelog.shand two bats suites.All five files
promote-rc.yaml's preflight demands are present now with their markers, where this branch previously satisfied one of five. The e2e job name is kept byte-identical because promote-rc matches on exactly that string, andhack/promote-gate-contract.batsnow pins the pair so the next drift gets a red test instead of a silent bypass.One deviation worth knowing about. The verify job is keyed on author plus a
release-head branch, not on the presence of thereleaselabel. On #3550 the PR was created at 09:35:31Z and the label arrived as a separatelabeledevent at 09:35:32Z, and this branch'son.pull_request.typeshas nolabeled, so a label-keyed job would satisfy the preflight marker and then never fire. Addinglabeledwas the other option but on main it is interlocked with a concurrency split, aplanguard complement ande2e-fork.yaml, so keying on the author is the narrower change. Checked against every release-labelled bot PR back to v1.0.7, the bot's other release PRs usechangelog-v*heads so they do not collide.