Skip to content

ci(release): complete the candidate-aware promotion pipeline on release-1.6 - #3893

Merged
myasnikovdaniil merged 4 commits into
release-1.6from
ci/release-machinery-release-1.6
Aug 18, 2026
Merged

ci(release): complete the candidate-aware promotion pipeline on release-1.6#3893
myasnikovdaniil merged 4 commits into
release-1.6from
ci/release-machinery-release-1.6

Conversation

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

v1.6.1 was promoted with the rc e2e gate bypassed, and its promote PR body says so verbatim. The reason is mechanical rather than anyone's judgement call: workflows run from the ref they fire on, release-1.6's tags.yaml has no rc-e2e job and this branch has no e2e-tag.yaml at all, so when v1.6.1-rc.1 was pushed there was nothing to produce evidence the gate could find. Bypass was the only outcome available.

This closes that. Adds e2e-tag.yaml verbatim from main, which works as written here because hack/e2e-chainsaw/ and the test-chainsaw target already exist on this branch. Splices main's rc-e2e job into tags.yaml. Adds the verify-release-candidate job to pull-requests.yaml, and the candidate-verification step to pull-requests-release.yaml positioned before tag creation, publish and retag. Plus hack/verify-promoted-packages.sh, hack/lib/promoted-packages.sh, hack/validate-changelog.sh and two bats suites.

All five files promote-rc.yaml's preflight demands are present now with their markers, where this branch previously satisfied one of five. The e2e job name is kept byte-identical because promote-rc matches on exactly that string, and hack/promote-gate-contract.bats now pins the pair so the next drift gets a red test instead of a silent bypass.

One deviation worth knowing about. The verify job is keyed on author plus a release- head branch, not on the presence of the release label. On #3550 the PR was created at 09:35:31Z and the label arrived as a separate labeled event at 09:35:32Z, and this branch's on.pull_request.types has no labeled, so a label-keyed job would satisfy the preflight marker and then never fire. Adding labeled was the other option but on main it is interlocked with a concurrency split, a plan guard complement and e2e-fork.yaml, so keying on the author is the narrower change. Checked against every release-labelled bot PR back to v1.0.7, the bot's other release PRs use changelog-v* heads so they do not collide.

…se-1.6

Promote RC refuses to promote a release whose target base cannot verify
the packages candidate. It checks five markers on the base branch before
its first registry write and fails closed with "Target base
'release-1.6' lacks '<path>'". This branch satisfied one of the five
(hack/lib/image-refs.sh), so the next patch release off release-1.6
could not be dispatched at all.

Add the two halves the base has to carry:

  * hack/verify-promoted-packages.sh and hack/lib/promoted-packages.sh,
    byte-identical to main. The verifier pulls the candidate by its
    committed digest and proves it matches the merge tree apart from the
    artifact's impossible self-reference, that promotion rewrote every
    version-line image tag, and that the container repo@digest set is
    unchanged from the rc artifact.

  * the two jobs that run it. verify-release-candidate in
    pull-requests.yaml checks GitHub's prospective merge tree so a
    packages/ edit pushed to the promote PR shows up as a failed check,
    and a Verify stable packages candidate step in the finalize job of
    pull-requests-release.yaml repeats it against the real merge commit.

Position is the whole point of the finalize half: it runs before the
write-once stable tag, the draft-release publish and the retag, so a bad
candidate is refused while nothing stable-named exists yet. The former
"Set up toolchain (skopeo, yq, helm)" and "Login to registry (GHCR)"
steps moved up with it and are no longer duplicated later in the job;
the toolchain now also installs flux and pins yq by version and sha256,
because an irreversible step must not depend on whatever
releases/latest/download resolves to today.

verify-release-candidate is keyed on the promote PR's author and its
head-branch shape rather than on the `release` label main matches. This
branch's on.pull_request.types has no `labeled` event, and the label is
applied about a second after the PR opens (a separate call in
gh pr create), so the `opened` payload carries no labels and a
label-keyed guard would never fire here.

.release-tooling is gitignored, matching main, so the trusted-base
checkout can never be staged.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Workflows run from the ref they fire on, so a tag pushed on this branch
executes THIS tags.yaml, not main's. main grew an rc-e2e job on
2026-07-24; release-1.6 never did. The v1.6.1-rc.1 push therefore
produced only Prepare Release, a skipped changelog and skipped website
docs — no E2E Release Candidate job at all. Promote RC looks for its
evidence by exact job name, found none, and v1.6.1 was promoted with the
gate bypassed. That is the hole this closes.

Add e2e-tag.yaml byte-identical to main and call it from a new rc-e2e
job. The suite step works as written here because this branch already
carries hack/e2e-chainsaw/ and a test-chainsaw target, and the sandbox
image already installs chainsaw 0.2.15.

Two things make the wiring load-bearing rather than cosmetic:

  * the called job's name must stay "E2E ${{ inputs.tag }} (full suite)".
    Promote RC composes the string it searches for from the rc tag, and
    GitHub renders a called job as "<caller> / <called>", which the
    gate's suffix matcher accepts. Rename either side and the gate fails
    closed.

  * the caller must grant contents: read AND checks: write. A caller's
    permissions are the ceiling for every job in the called workflow and
    GitHub validates that ceiling statically, at run creation, before any
    `if:` is evaluated. e2e-tag.yaml's e2e job declares checks: write for
    a dispatch-only breakpoint workflow_call can never reach, so omitting
    the grant would fail the entire tags.yaml run for EVERY tag push, rc
    and stable alike, building nothing.

prepare-release already publishes the rc as a non-draft prerelease with
the nocloud-amd64.raw.xz asset and pushes the release-X.Y.Z-rc.N staging
branch before rc-e2e can start, which is exactly what e2e-tag.yaml's
resolve job requires.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Since the promote PR is based on release-1.6, the changelog it carries
merges onto the maintenance line and never reaches main. The backstop job
in tags.yaml only asks whether the file is on origin/main, so it sees
"absent", spends an AI run, and produces text nobody reviewed — which
update-releasenotes.yaml then pushes into the published release body,
overwriting the reviewed notes the release actually shipped with.

Ask the second question too. `at_tag` is whether the changelog is present
in this checkout, which is the tag commit and therefore the promote merge
commit. When it is, port that reviewed file to main verbatim and skip
generation entirely; the AI now runs only when the changelog is genuinely
absent on both sides.

Add hack/validate-changelog.sh, byte-identical to main, and gate the
commit on it. A ported file came from the tag and a generated one came
from a step that is deliberately allowed to fail, so neither had been
checked: the old guard only asked whether the file was non-empty, which a
truncated Copilot stream satisfies. The validator asserts the H1, the
release-link comment, the trailing compare link and at least one section,
all carrying the expected version, so a fragment or a changelog generated
against the wrong tag cannot reach a release body.

Verified against this branch's own shipped changelogs: v1.6.1, v1.5.2 and
v1.5.1 all pass, and v1.6.1 checked against 1.6.2 fails as it should.

This takes effect for tags cut after it lands — tags.yaml and the script
it calls both come from the tag's own tree.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/verify-promoted-packages_test.bats is byte-identical to main: 13
tests driving the verifier against a mocked `flux`, so they need no
registry. They cover the fail-open shapes that matter most — an
unreadable file during the rc-reference scan, a collector that returns
nothing, a digest that moved while candidate and merge tree still match.

hack/promote-gate-contract.bats is NOT main's copy, and the difference is
deliberate. main's version pins a `parse` job, a `skip_e2e_gate` input,
promote-time changelog and website-docs jobs, and the `labeled`
pull_request trigger they depend on — none of which exist here, because
`Promote RC` is a workflow_dispatch and runs from the ref it is
dispatched from, which is main. Run verbatim against this branch it fails
on its first test and the runner aborts the suite.

So this pins the half release-1.6 actually owns — the producer side:

  * an rc tag push calls e2e-tag.yaml with the pushed ref, only for
    -rc. tags, and grants the static permission ceiling the called
    workflow declares;
  * the called job's name is the exact literal main's gate composes and
    searches for, asserted with grep -F because `${{` as a basic regular
    expression matches nothing and would make the pin pass by finding its
    own subject absent;
  * the suite it runs is the full one (empty CHAINSAW_SUITES), with the
    Makefile target and the sandbox chainsaw binary it needs;
  * all five files promote-rc.yaml's preflight reads off the target base
    are present, plus the two libraries the verifier sources by relative
    path;
  * verify-release-candidate is keyed on author and head branch and
    carries NO label condition, so restoring main's from muscle memory
    switches the job off here and fails this test instead;
  * the candidate verification precedes the stable tag, the submodule
    tag, the release publish, the retag and the chart publish, and
    follows the toolchain, the login and the tooling checkout it needs;
  * the tag-time changelog is validated and ported, not regenerated.

Every pin was mutation-checked: breaking the thing it describes turns it
red.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 79325cf5-b17c-44ef-a08e-857cf0c6c895

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) size/XXL This PR changes 1000+ lines, ignoring generated files labels Aug 18, 2026
@myasnikovdaniil
myasnikovdaniil merged commit f6f1861 into release-1.6 Aug 18, 2026
40 of 41 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the ci/release-machinery-release-1.6 branch August 18, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant