chore(release): promote v1.6.0-rc.4 -> v1.6.0 - #3397

…rror (#3404)

## The bug

Promotion rewrites the rc version substring to the stable version across
the vendored image references. That rewrite lived inline in
`promote-rc.yaml` and globbed the depth-2 package `values.yaml` plus
`packages/apps/kubernetes/images/*.tag` **only**, on the premise that
the kubernetes app was the only one whose `.tag` files carry the
cozystack version. Nine other `.tag` files do, and `system/multus` seds
its reference straight into a vendored upstream daemonset manifest.

#3397 (promote `v1.6.0-rc.4` → `v1.6.0`) is therefore staged with **33
of 55 first-party references still reading `v1.6.0-rc.4`**.

`hack/promote-retag.sh` and `hack/nightly-mirror.sh` had the same blind
spot from the other direction — both scanned the depth-2 `values.yaml`
alone, so those images were never retagged to the stable version (**30
references selected before, 42 after**) and never mirrored to the public
registry for a nightly. A retag miss leaves an image short of a tag,
since the digest still resolves inside one registry. A mirror miss is
worse: the host rewrite walks the same file list, so a reference that is
neither mirrored nor rewritten leaves the published tree pointing at the
private build registry.

`v1.6.0` is the first release to go through promotion — every earlier
release was a full rebuild from the tag, which stamped every file
uniformly. The glob has always been wrong; the old path just made it
unobservable.

## Why it could not be caught before

The rewrite was workflow-inline, so there was nothing to unit test. The
only way to observe the miss was to cut a release.

It is a script now — `hack/promote-rewrite-tags.sh` — and
`hack/promote-rewrite-tags_test.bats` round-trips it against the real
tree under `make unit-tests`, with no cluster, registry or release
involved.

## The fix

The three call sites each carried their own idea of where a reference
can live, which is how they drifted apart. That knowledge moves into
`hack/lib/image-refs.sh` as the single enumeration — `image_ref_files()`
for the files, `collect_image_refs()` for the references inside them —
and all three source it, so a new storage location is declared once and
reaches every consumer.

The rewrite also asserts its own postcondition: it scans **wider** than
it rewrites and fails the promotion when a reference survives in a
location the enumeration does not know about, rather than shipping it
quietly. The wide scan is shape-filtered to image-reference position,
because `X.Y.Z-rc.N` is an ordinary version string that other things
legitimately contain — `kubevirt-csi-driver/go.sum` pins
`github.com/golang/protobuf v1.4.0-rc.2`, and `v1.4.0-rc.2` is a
cozystack rc that was actually cut. Narrowing it to `${RC}@sha256:`
instead would have been wrong: `system/cilium/values.yaml` keeps `tag:`
and `digest:` under separate keys.

`docs/agents/image-refs.md` records the contract — three tag classes,
three storage shapes, the invariants, and which consumer does what.

## Not a bug: kamaji

`cluster-api-control-plane-provider-kamaji` stays at
`v0.19.0-cozystack.0` across the promotion. It is a first-party rebuild
of an upstream component, versioned by that component rather than by the
cozystack version line, so no rewrite is owed. A test pins this so a
future fix here cannot over-reach.

## Documented known gaps, deliberately not fixed here

Both are pre-existing and in the **host** dimension; the version
dimension this PR owns is complete, because the version always lives in
a `tag` key regardless of where the host sits.

- **The nightly host rewrite cannot reach a split host.** It is a
literal `<src-registry>/` substring replace, so `keycloak-operator`
(host in a sibling `registry:` key) and `kubeovn`
(`global.registry.address`, no trailing slash) are mirrored but keep
pointing at the build registry in the published tree. Fixing it needs
structure-aware rewriting.
- **A reference inside a gzip is invisible.**
`capi-providers-cpprovider` ships `files/components.gz`, and the promote
postcondition cannot catch this class at all since `grep -rIl` skips
binaries. Nothing is owed today — kamaji is component-versioned and the
image is still mirrored and retagged via its `.tag` — so the residual is
that a nightly's kamaji ConfigMap keeps the build-registry host.

## Testing

- 17 tests in the new suite; `make bats-unit-tests` green (312 tests,
exit 0); `shellcheck -x` clean; `zizmor` clean on the workflow.
- Verified against four real rc versions (`1.4.0-rc.2`, `1.4.0-rc.4`,
`1.5.0-rc.2`, `1.6.0-rc.4`) in both directions — no false positives, no
false negatives.
- Every test is mutation-proven. Reverting the enumeration to the old
glob, deleting any of the five YAML shapes, corrupting emitted digests,
reverting extras to parse-only, dropping the unreadable-file guard,
broadening shape 4 beyond `global.images`, or planting an unenumerated
first-party reference in any storage shape each fail a named test.

## Reviewer note, out of scope

`platform-migrations` is pinned at two different digests —
`core/platform/values.yaml` at `v1.5.0` and
`backupstrategy-controller/values.yaml` at `v1.4.0-rc.2` (an unowned
pin; no Makefile writes that key, tracked in #3143). A
`promote-retag.sh` dry-run emits **two copies to the same destination
tag**, so the write-once guard will fail the promotion. This is
independent of this PR and needs a decision about the backup-client
image version, so it is left alone here — but it blocks the 1.6.0
promote and wants fixing before the re-cut.

## Release note

```release-note
Promotion, retagging and nightly mirroring now cover every image reference the build stamps, including those stored in `images/*.tag` files and in vendored manifests. Previously a promoted release could ship images still tagged with the release-candidate version, and those images received no stable tag in the registry.
```

Fixes the promote half of #3143.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Image promotion and nightly mirroring now detect digest-pinned image
references across values files, standalone tag files, and supported
templates.
- Stable release preparation supports RC→stable tag rewriting via a
dedicated promotion script, with broader verification to ensure no RC
references remain.

- **Bug Fixes**
- Improved completeness of ref discovery and scanning, including
previously missed template/tag locations.

- **Documentation**
- Added and linked guidance documenting image-reference invariants and
how tooling enumerates supported storage shapes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

)

## What this PR does

The promote step guarded PR creation with `gh pr view <branch>`, which
resolves a pull request by head branch **in any state**. A closed one
satisfied the guard, so creation was skipped while the step still exited
0 — a promotion that reports success and leaves nothing to merge. Same
silent-skip shape as the enumeration bug fixed in #3404: the run is
green, the artifact is missing, and nothing says so.

`STABLE_BRANCH` is derived from the stable version
(`release-${version}`), not from the rc number, so every rc promoted to
that version shares it. Once any promote PR for a version has been
closed, both re-dispatch paths this workflow documents as supported are
wedged: re-dispatching the same rc, and promoting a newer rc to the same
version. That second one is named in the workflow's own header as the
reason a leftover draft is tolerated — the tolerance was implemented for
the release draft and missed for the PR.

Neither path recovers by reopening the old PR, because the step's own
`git checkout -B` plus force-push makes that PR's head unreachable, and
GitHub then refuses:

```
422 Validation Failed
  state cannot be changed. The release-1.6.0 branch was force-pushed or recreated.
```

The fix asks the question the guard meant to ask — is there an **open**
PR for this head — and creates one otherwise.

**Hit live.** Promoting `v1.6.0-rc.4` after #3397 had been closed: [run
29917852639](https://github.com/cozystack/cozystack/actions/runs/29917852639)
went green, logged `PR already open for release-1.6.0`, and opened no
PR. This currently blocks the v1.6.0 release.

**Verified in both directions** against live data, so the change is not
vacuous:

| head branch | PR state | old guard | new guard |
| --- | --- | --- | --- |
| `release-1.6.0` | #3397 closed | `TRUE` → skip (bug) | `FALSE` →
create ✅ |
| `chore/gitignore` | #3412 open | `TRUE` → skip | `TRUE` → skip ✅ |

`actionlint` exits 0 and `zizmor` reports no findings. The guard reads
`${STABLE_BRANCH}` as a shell variable from the step's `env:` block
rather than as a `${{ }}` expansion inside `run:`, so no
expression-injection surface is added.

### Screenshots

N/A — no UI change.

### Downstream repositories

Walked the trigger map in `docs/agents/contributing.md` against the
diff, which is one file: `.github/workflows/promote-rc.yaml`. The only
workflow-related trigger in the map is cozystack/ccp on "change
release-prep behaviour in `.github/workflows/tags.yaml`", which this
does not touch. This change also restores the documented behaviour of
`promote-rc.yaml` rather than altering its contract, so nothing
downstream sees a result different from what the docs already promise.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(release): promoting a release candidate now opens the promotion pull request even when an earlier promotion attempt for the same version was abandoned. Previously the workflow mistook a closed pull request for an open one, skipped creating a new one, and reported success with nothing left to merge.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved release PR creation checks so closed or previously merged
pull requests no longer prevent new release PRs from being opened.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

The promote step guarded PR creation with `gh pr view <branch>`, which
resolves a pull request by head branch in any state. A closed one
satisfied the guard, so creation was skipped while the step still exited
0 — a promotion that reports success and leaves nothing to merge, the
same silent-skip shape as the enumeration bug fixed in cozystack#3404.

STABLE_BRANCH is derived from the stable version, so every rc promoted to
that version shares it. Both re-dispatch paths this workflow documents as
supported are therefore wedged once any promote PR for that version has
been closed: re-dispatching the same rc, and promoting a newer rc to the
same version. Neither recovers by reopening the old PR, because the
step's own force-push makes its head unreachable and GitHub refuses with
"state cannot be changed. The <branch> branch was force-pushed or
recreated."

Hit live while promoting v1.6.0-rc.4 after cozystack#3397 had been closed: the run
went green and opened no PR.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Now that the required "E2E Tests" context is a commit status rather
than a job, a run that dies before posting leaves the head SHA at
"Expected" — which blocks the merge instead of passing it. Two paths
reached that state.

Same-repo: a `labeled` event shared the PR's concurrency group, so
labelling an open PR cancelled the live opened/synchronize run, and
the replacement run was then discarded by `plan`'s full-e2e guard --
so `e2e-report` skipped with it and nothing posted the status the
cancelled run was going to. Observed on live CI: run 30614258694
(cancelled) followed by 30614280603 (skipped) on the same SHA, 24s
apart, from a maintainer adding kind/* labels just after a push.
Label events now get their own concurrency group, so they can only
supersede each other. Release promotion is unaffected: `gh pr create
--label` attaches `release` at creation, so a promote PR gets a
single run that already sees the label (verified on #3397 and #3425,
one run each, build/finalize gated off).

Fork: `resolve` had no equivalent, so a cancelled triggering run fell
through to the `CONCLUSION != 'success'` branch and posted `failure`
— a false red on a SHA whose e2e never ran, which the replacement run
for that same SHA would not flip back (it takes the discarded-label
path and posts nothing). It now returns early on a cancelled or
wholly-skipped conclusion. Checked on the conclusion rather than the
job list because a cancelled run can report no jobs at all, which
would make the `Plan build` lookup miss.

Both stay fail-closed: declining to post can only preserve a verdict
a real run reached, never invent one.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>