fix(ci): overlay images from the PR base branch, and publish per-line artifacts - #3471

Actionable comments posted: 1

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/overlay-main-images_test.bats`:
- Around line 157-177: Harden the wiring test around the overlay step loop so it
validates a standalone YAML if: key rather than any matching text in the step
block. Add a mutation check for each step that removes its base-branch guard and
asserts the test fails, ensuring the guard is structurally required and cannot
be satisfied by run content or comments.

LGTM. The if: github.base_ref == 'main' guard targets the root cause, not a symptom: the current-main overlay's premise (committed refs are stale, repoint them at main digests) only holds for main-based PRs. On release-line PRs it was deploying main binaries onto release charts — the failure mode #3437 hit.

Verified: pull_request trigger always populates base_ref; release branches use the release-* naming so == 'main' is an exact discriminator; _out/mainpkgs is consumed only by these two steps, so release-line PRs skip both cleanly with no dangling refs. bats 12/12 green incl. the new guard test (mutation-checked). No regressions — main-based behaviour is byte-identical.

Actionable comments posted: 2

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/build-release.yaml:
- Around line 69-72: Update the “Set up build toolchain” step to stop piping the
mutable fluxcd.io installer directly into sudo bash; download a version- or
commit-pinned Flux installation asset, verify its SHA-256 checksum against a
pinned expected value, and only then execute it, preserving the existing
command-availability check.

In @.github/workflows/pull-requests.yaml:
- Around line 405-409: Update the BASE_REF == "main" branch in the
missing-artifact handling to emit the same GitHub Actions ::warning annotation
as the release-line branch, while retaining the existing context that committed
refs are being kept. Ensure missing artifacts on main trigger a workflow warning
rather than stdout-only logging.

LGTM — the base-branch resolution is real code and not a behaviour pinned only by tests, the two new wiring tests actually go red when the thing they protect is removed, and nothing here changes what gets published to the release registry.

Business context: the PR-finalize overlay could only ever read the main packages artifact, so a PR based on a release line was handed main's binaries to run against that line's charts; this makes the overlay read the PR's own base branch and adds the workflow that publishes the per-line artifact it needs.

What I checked

The problem reproduces from source on main, not just from the description. The pull_request trigger carries no branch filter, and the finalize step pulls cozystack-packages:main unconditionally before overlaying it onto every package the PR did not rebuild. So a release-line PR gets the cross-generation mix — and the quieter half of that is that a green e2e on such a PR proves nothing about the line it targets.

The base-branch resolution is implemented, not merely asserted. pull-requests.yaml pulls cozystack-packages:${BASE_REF} with BASE_REF: ${{ github.base_ref }} wired into that step's env. hack/overlay-main-images.sh is untouched and does not need to change: it consumes an extracted directory and never resolves a branch itself. The producer side closes the loop — build-release.yaml sets IMAGE_TAG: ${{ github.ref_name }}, the root build target ends in make -C packages/core/installer image, and that Makefile's image-packages pushes oci://$(REGISTRY)/cozystack-packages:$(IMAGE_TAG).

The changed step did execute on this head: the finalize job logged pulling artifact from …/cozystack-packages:main followed by overlaid=4 same=127 skipped=40 drift=0 failed=0, so the main path is proven by a real run and not only by a YAML assertion. Neither the release-line path nor build-release.yaml is exercised by any PR run — the latter is push-triggered on a line branch, so it cannot run until it is on one. That is worth stating plainly since the description already acknowledges the branch filter as locally unprovable.

I mutation-checked the two new tests rather than trusting the green. Reverting the tag to :main, dropping BASE_REF from that step's env, renaming the step, broadening the trigger to release-*, flipping WRITE_CACHE to '1', pointing IMAGE_TAG away from github.ref_name, and deleting build-release.yaml each turn the file red with a TAP not ok and 12 of the declared 13 ok lines — under plain bats and under hack/cozytest.sh, which is what make unit-tests runs. These are not assert-the-guard-exists tests.

The branch filter is the same glob shape GitHub documents for a version pattern: the character class matches one digit, + repeats it, . is literal, and the match is anchored to the whole ref. It selects release-1.6 and rejects release-1.6.1 and release-1.6.0-rc.4, all three of which exist on the remote right now.

The backport label is applicable as written: the squashed diff applies cleanly against the current release-1.6 tree, and that branch's build-main.yaml, hack/common-envs.mk and installer Makefile are identical in every part this depends on, including the runner label and the artifact push. The first merge into the line bootstraps the line's own artifact, so there is no chicken-and-egg step to perform by hand.

Merging this to main alone fixes nothing — a push workflow runs the copy present on the pushed branch, and a pull_request run uses the base branch's copy — so the backport is the delivery, not a follow-up. I do not treat that as a blocker: it is stated in the description, the label that automates it is already on the PR, and the new annotation is what surfaces the gap if the rollout stalls.

Blast radius on release engineering is nil. Everything here pushes to the CI registry; stable publishing runs against the public registry from the release finalize path, which this does not touch.

Non-blocking follow-ups

1. The naming this change invalidates deserves a sweep. hack/overlay-main-images.sh still names the main artifact as its source of truth in both the header and the usage block, still prints Overlay current-main images: and -> current-main per file, and docs/agents/image-refs.md still describes the tool as overlaying "current-main references". This PR's own finalize log shows overlay: packages/system/seaweedfs/values.yaml -> current-main; on a release-line PR that same line will say "current-main" while overlaying that line's refs. I am not blocking on descriptive text and the corrective paragraph is right there in the workflow — but the incident being fixed here was itself a misdiagnosis, so a log line that lies about which generation was applied is the wrong thing to leave behind.

2. The comment this diff added on the overlay step says the script handles "the empty directory the step above leaves when a line has no artifact yet". The step does rm -rf _out/mainpkgs on failure, so it leaves no directory at all, and the script's guard is the missing-directory branch. Behaviour is right, the comment is not, and it is a comment this diff wrote. Related: the paragraph directly above the changed step still asserts :main as the source and is contradicted by the new paragraph under it.

3. Trigger breadth against cost. paths-ignore: ['docs/**'] means any non-docs push to a maintained line fires the full build — a workflow edit, a test-only change, a merged promote PR. On main that breadth pays for itself because the run warms the shared cache; here WRITE_CACHE is '0', so a run that produces no new image is pure cost on a large runner. Narrowing to the paths that can actually change an image would remove most of it without weakening the guarantee.

4. On a fork PR the registry login step is skipped, so flux pull fails for want of credentials. Against main that is the quiet echo, but against a release line it now emits "Check that build-release.yaml has run for …", which points at the wrong cause. Distinguishing "no credentials" from "no artifact" costs one condition.

5. build-release.yaml is close to a copy of build-main.yaml and nothing pins the two in sync, so a future fix to the toolchain or buildx setup in one silently misses the other. Same for the missing repository guard: a fork that syncs a release line triggers a large, long job that dies at login with empty secrets. Both files share that gap, so this mirrors the house pattern rather than introducing one.

6. The ::warning on a missing artifact is presented as a deliberate design choice, but nothing pins it — a later edit can drop it silently, which is the same class of gap the rest of this change closes.

7. The two open review-bot comments still stand. On pinning the flux installer in particular: this job holds registry push credentials and the artifact it produces becomes the image source for every PR on that line, so an unpinned installer piped into sudo bash matters more here than its age in build-main.yaml suggests. If it gets fixed, fixing both files in one pass is the cheaper move.