ci(release): port the checks-at-rc release machinery to release-1.5 - #3894

The checks-at-rc gate correlates its evidence by workflow job name:
promote-rc.yaml looks for `E2E ${rcTag} (full suite)`, which is what
e2e-tag.yaml's e2e job publishes. release-1.5 had no such workflow, so
the only route to promotion would have been skip_e2e_gate=true -- the
exact bypass the model exists to make unnecessary. tags.yaml's rc-e2e
job (next commit) calls this file for every vX.Y.Z-rc.N tag.

Ported from main with ONE adaptation, the suite step. main runs

  make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME \
    test-chainsaw CHAINSAW_SUITES=""

which cannot work here: packages/core/testing/Makefile defines only
test-apps-%, hack/e2e-chainsaw/ does not exist, and the e2e-sandbox
image installs no chainsaw binary. In its place the step runs the bats
loop over every hack/e2e-apps/*.bats, collecting failures and exiting
non-zero once at the end -- the shape release-e2e.yaml and this branch's
PR lane already use. Its per-app diagnostics go through
`docker exec "$SANDBOX_NAME"` rather than the bare `kubectl` that
release-e2e.yaml used: the cluster lives inside the sandbox container
and the runner holds no kubeconfig for it, so the bare form printed
nothing.

Everything else is main's, byte for byte -- the workflow_call and
workflow_dispatch interfaces, the resolve job's release/asset/tree
validation, the runner shape, the collect-report ceiling and its overrun
warning, the breakpoint step, and above all the job NAME. main's two
chainsaw-report steps are dropped; they copy a path this branch never
produces.

release-e2e.yaml is deleted in its place. Its trigger was

  on:
    push:
      tags:
        - 'v*.*.*'
        - 'v*.*.*-rc.*'
        - 'v*.*.*-beta.*'
        - 'v*.*.*-alpha.*'

byte-for-byte the surface tags.yaml already claims, so every tag push
started both. It ran a second full `make build PUSH=1` with no IMAGE_TAG
set -- republishing every image under the shared :dev tag -- plus a
second 120-minute bats suite, and its job was named `Release E2E (full
suite)`, which the gate can never match. It has been red on every 1.5
tag.

hack/select-e2e.sh follows in the same commit: the workflow term becomes
e2e-tag rather than release-e2e, and hack/lib/ is added because
hack/lib/*.sh does not match the existing hack/[^/]+\.sh$ term and would
otherwise fall through to "silently ignored", selecting no e2e at all
for a change to the shared image-reference enumeration.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

The 1.5 line still carried the pre-#3449 release machinery: tags.yaml
built and opened a release PR for any tag including a hand-pushed stable
one, finalize force-moved the tag and published the draft with no
candidate verification and no registry promotion, and there was no way
to cut a pre-release except by pushing a tag by hand. Meanwhile
promote-rc.yaml on main refuses to start unless the target base branch
carries five specific files, none of which were here -- so a `Promote RC`
dispatch for v1.5.4-rc.1 failed at `parse`.

Workflows run from the ref they fire on, so porting these files onto the
branch is what makes the model real for this line rather than a property
of main.

  .github/workflows/tags.yaml
      Replaced with main's. One value adapted: CACHE_REGISTRY becomes
      iad.ocir.io/idyksih5sir9/cozystack-cache-release-1.5, because this
      branch's own cache warmer (build-main.yaml, "Build cache
      (release-1.5)") writes there and main's refs carry main's sources.
      The diff against main is exactly that line plus its comment.
      What this brings: rejection of hand-pushed stable tags, rc releases
      published rather than left draft, a compare-before-force staging
      branch, the mandatory rc-e2e job, the changelog backstop's
      port-from-tag path and its validate-changelog.sh gate, and the
      `Set up build toolchain` flux install that ephemeral runners need
      because packages/core/installer's image-packages target shells out
      to `flux push artifact` -- the old workflow only got away without
      it by pinning runs-on: [self-hosted].
      `Get base branch` accepts release-1.5 unchanged:
      /^release-\d+\.\d+$/.

  .github/workflows/pull-requests-release.yaml
      Replaced with main's, byte-identical. Finalize now verifies the
      packages candidate from the trusted base SHA BEFORE creating any
      write-once name, cuts tag and Go-module tag write-once instead of
      force-moving them, fast-forwards the maintenance branch instead of
      force-updating it, takes the release body from the changelog in the
      merge commit, and performs the registry promotion (retag by digest,
      :latest only for the newest stable, stable cozy-installer chart).
      Note release-1.6's copy is NOT byte-identical to main's any more --
      main gained the candidate-verification layer after 1.6 was cut --
      so this branch is now the closer of the two to main.

  .github/workflows/pull-requests.yaml
      Surgical: the `verify-release-candidate` job only. main's copy is
      ~1297 lines and depends on plan/build-matrix.sh/select-install.sh/
      chainsaw TIA, and publishes `E2E Tests` as a commit status from
      e2e-report, whereas this branch's ~469-line copy uses `E2E Tests`
      as the job name of its e2e job -- replacing the file would break
      the required check. The job is self-contained: prospective-merge
      checkout, `.release-tooling` from the base SHA, pinned flux/yq,
      GHCR login, then .release-tooling/hack/verify-promoted-packages.sh.
      Its guard is NOT main's, and that is the point. main keys the job
      on the `release` label, which it can because its
      on.pull_request.types carries `labeled`; this branch's carries only
      opened/synchronize/reopened. `gh pr create --label release` is two
      API calls -- PR #3550 was created 09:35:31Z and labelled 09:35:32Z
      -- so a label-keyed guard here fires only when GitHub happens to
      serialize the `opened` payload after the labelling call. It has
      been winning that race on this branch (PR #3161: opened 01:58:36Z,
      labelled 01:58:37Z, and the `opened` run created at 01:58:40Z did
      execute the label-gated `Resolve assets`), which is the dangerous
      outcome rather than the safe one: the job would look green while
      being one scheduling hiccup away from silently not running, and it
      would satisfy promote-rc.yaml's five-file preflight either way.
      Keyed instead on author plus head-branch shape, which is fixed at
      PR creation. Checked against every `release`-labelled PR this bot
      has opened, back to v1.0.7: a `release-*` head from
      cozystack-ci[bot] is always a release PR, and its only other
      `release`-labelled head shape is `changelog-v*` (#2971), excluded
      by startsWith. Prerelease heads are excluded explicitly -- #2794,
      #2924 and #2831 show the old flow produced them, and
      ${HEAD_BRANCH#release-} must be X.Y.Z.
      Backporting main's `labeled` trigger is deliberately NOT the fix:
      on main that key is interlocked with a concurrency-group split, a
      `plan` guard complement and e2e-fork.yaml.

  .github/workflows/cut-prerelease.yaml
      Verbatim from main. All branch logic is generic regex. The freeze
      step is gated on kind == 'rc' && patch == '0' and the build-release
      dispatch on that step's `created` output, so neither fires for a
      patch cut from this branch -- and the one tag shape that would
      reach the freeze, v1.5.0-*, is already taken and refused by the
      write-once guard. NOTE build-release.yaml does not exist on this
      line; that dispatch is unreachable here and deliberately left as
      main has it rather than removed.

Two contract suites come with it, trimmed to what this branch owns:

  hack/promote-gate-contract.bats     main owns the gate's asking side
                                      (promote-rc.yaml is dispatched from
                                      main); this pins the answering
                                      side -- the job name e2e-tag.yaml
                                      publishes, the permission ceiling
                                      rc-e2e must grant, finalize's
                                      persist-credentials:false and its
                                      verify-before-write-once ordering,
                                      that the adapted suite step calls
                                      a target the Makefile has, and that
                                      the guard above never drifts back
                                      to a label key
  hack/release-freeze-contract.bats   the cut-prerelease freeze
                                      invariants; main's copy also covers
                                      backport.yaml and build-release.yaml,
                                      neither portable here

Every pin was mutation-proven, by performing the violation rather than
by reverting the fix: renaming the e2e job name, dropping `checks: write`
from either side of the rc-e2e/e2e-tag pair, dropping
persist-credentials:false from finalize's checkout, moving the candidate
verification after the write-once tag, disabling cut-prerelease's
dispatch-branch guard, and -- for the guard above -- regressing it to
main's label key, dropping the author term, dropping one prerelease
exclusion, and running the verifier from the head instead of the base.
All ten turn the relevant test red.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Its schedule half is already dead -- GitHub runs `schedule` only from the
default branch, and main removed the workflow -- but the dispatch half is
still reachable from release-1.5:

  on:
    schedule:
      - cron: '0 1 * * *'
    workflow_dispatch: # Allow manual trigger

and what it does when reached is delete and re-point a stable tag:

  execSync(`git push origin :refs/tags/${nextTag}`, ...)
  execSync(`git tag ${nextTag} ${latestBranchCommit}`, ...)
  execSync(`git push origin HEAD:refs/tags/${nextTag}`, ...)

`nextTag` is derived from the newest PUBLISHED release on the line, which
for 1.5 is v1.5.2, so nextTag is v1.5.3 -- a tag that already exists,
whose GitHub release is a draft and whose promote PR closed unmerged. One
dispatch deletes and re-creates it. Its run history already shows it
re-pushing v1.5.2 on 07-01/02/03 and v1.5.3 twice on 07-07.

That is incompatible with everything the previous commits establish: the
whole model rests on stable tags being write-once, cut only by finalize
at a promote PR's merge commit, and on tags.yaml refusing a stable tag
that has no pre-existing draft. Under those rules this workflow's output
is a tag tags.yaml rejects, pushed over one that is already published.

There is no replacement: a patch is now cut deliberately with `Cut
Pre-release Tag` for vX.Y.Z-rc.N and promoted, which is the point.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

A `release`-labelled PR entered the e2e job through resolve_assets: the
label makes that job run, it finds the draft release promote-rc created,
and `always() && needs.resolve_assets.result == 'success'` was enough to
start e2e. `Select E2E tests` then saw the digest rewrite under
packages/core/ and escalated to every hack/e2e-apps/*.bats file.

So every promote PR burned hours of a 24-core runner re-testing container
digests that are bit-for-bit the ones e2e-tag.yaml already ran the full
suite against -- and which promote-rc.yaml refuses to promote without
that green evidence. Measured on this line's own release PRs: #3219
(v1.5.3) ran `E2E Tests` for 1h12m51s, and #3161 (v1.5.2) for 1h41m59s
before failing. docs/release.md already said E2E does not run on the
promote PR by default; it did.

The release arm now also requires the `full-e2e` label, which is the
shape main uses. Chosen over a flat `&& !contains(..., 'release')`
because it keeps an opt-in door rather than closing the lane outright,
and it converges with main so a later backport is a no-op here.

One caveat this branch does not share with main, recorded in the comment:
`on.pull_request.types` here has no `labeled`, so adding `full-e2e` to an
already-open promote PR does not start a run by itself -- it takes effect
on the next push to the branch, and a workflow re-run does not help
because the stored `opened` payload's label list is what the expression
reads. Backporting main's `labeled` trigger would fix that, but it also
needs main's concurrency-group split, without which a label event
cancels the live run and then skips, leaving the required `E2E Tests`
check cancelled and the PR unmergeable. Out of scope here. The button
that always works meanwhile is e2e-tag.yaml's workflow_dispatch, which
takes the tag and runs the same full suite against the published tree.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…t deploy time

Promotion never rebuilds: the stable image is the rc image, retagged by
digest. So the operator ships with VERSION=v1.5.4-rc.1 baked in by
ldflags, and v1.5.4 would report itself as v1.5.4-rc.1 in
cozy_cluster_info{cozystack_version=...}.

main corrects this at deploy time and release-1.5 had neither half of the
plumbing. pull-requests-release.yaml's finalize -- ported two commits ago
-- already runs

  yq -i '.cozystackOperator.platformVersion = "v" + strenv(STABLE_VERSION)' \
    packages/core/installer/values.yaml

on the working tree before packaging the stable chart, so without the
receiving end that stamp writes a key nothing reads.

Four files, all matching main:

  pkg/version/version.go       COZYSTACK_VERSION wins over the baked
                               value when set and non-empty
  pkg/version/version_test.go  the three cases (unset, empty, set)
  packages/core/installer/{values.yaml,templates/cozystack-operator.yaml}
                               platformVersion, rendered as the
                               COZYSTACK_VERSION env under a single
                               conditional interleaved with the
                               per-variant endpoint env
  Makefile                     the manifests target stamps it into the
                               kubectl-apply artifacts; that hunk is now
                               byte-identical to main's

One behavioural delta from adopting main's template block verbatim: the
hosted variant with no platformVersion now renders no `env` key at all
rather than `env: []`. Semantically identical to Kubernetes, and it is
what lets main's helm-unittest suite come across unchanged.

That suite needed a home: packages/core/installer/Makefile had no `test:`
target, so hack/helm-unit-tests.sh skipped the chart entirely. Added, and
main's tests/cozystack_version_test.yaml comes with it -- 5 tests, all
five variants x platformVersion combinations.

Verified end to end on this branch: the finalize yq stamp against
STABLE_VERSION=1.5.4 renders `COZYSTACK_VERSION: "v1.5.4"` on the
operator container. Red phase: deleting the template's COZYSTACK_VERSION
block takes 3 of the 5 helm-unittest cases red.

Sole consumer on this line is internal/telemetry/operator_collector.go,
so the blast radius is the telemetry label -- but a shipped release
reporting the wrong version string is worth fixing.

Note neither branch's `make unit-tests` runs ./pkg/version/... --
go-unit-tests is scoped to pkg/registry, pkg/config and pkg/cmd/server on
main too. Left as main has it rather than diverging; `go test
./pkg/version/...` was run by hand and passes.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…etag

Promotion retags by digest: hack/promote-retag.sh copies every collected
<repo>@<digest> to <repo>:<stable-version>. Two digests under one
repository means two copies aimed at one destination tag — the first
wins, the second hits the write-once guard, and the release fails at the
retag step, after the stable tag is pushed and the release is public.

platform-migrations was pinned twice on this line:

  packages/core/platform/values.yaml  .migrations.image
    platform-migrations:v1.5.2@sha256:5d601b07...
  backupstrategy-controller/values.yaml  .chBackupClientImage
    platform-migrations:v1.4.0-rc.2@sha256:17390197...

The second had no producer at all — its comment asked a human to keep the
two in lockstep — so it froze at v1.4.0-rc.2 while the first advanced
through v1.5.0/v1.5.1/v1.5.2. Every release so far was a full rebuild, so
nothing retagged and nothing noticed. v1.5.4 is the first cut through the
promote path.

Same fix main made: packages/core/platform/Makefile now stamps both keys
from the one digest it just built, and the second pin is realigned to the
ref that stamp would produce today. hack/image-pin-consistency.bats pins
the postcondition and the producer, red-phase checked by reverting the
value.

Kept as its own commit because it changes what a shipped 1.5.x renders
into the Altinity strategy Pod (the curl+jq runner moves from the
v1.4.0-rc.2 image to the v1.5.2 one), which is a functional change to a
maintenance line rather than part of the CI port.

This does NOT make the retag pass on its own — see the BLOCKED note on
ubuntu-container-disk, which holds six legitimate digests under one
repository on this line and is the other duplicate destination. This
commit removes the one duplicate that is a defect; that one is a
promotion-contract gap and is reported rather than patched here. main's
copy of image-pin-consistency.bats leads with the general "no repository
pinned at more than one digest" check for that reason; it is deliberately
not ported, since this branch's tree legitimately violates it.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…flow

docs/release.md was written for the model this branch just stopped
running, and two of its instructions are now actively wrong rather than
merely stale: "Always push tags with `git push origin
HEAD:refs/tags/<tag>`" (tags.yaml rejects a hand-pushed stable tag), and
a whole "Automated patch tag (`auto-release.yaml`)" section describing a
workflow that no longer exists. docs/agents/e2e-testing.md pointed twice
at release-e2e.yaml.

Corrected in place rather than rewritten. A banner at the top states the
six deltas, and the sections that contradicted a live workflow are fixed:
the workflow inventory now names cut-prerelease, e2e-tag, the
verify-release-candidate job and where promote-rc.yaml actually lives;
phase 1 no longer claims to open a release PR; phase 4 is the promote PR
and says E2E does not run on it by default; phase 5 is the real finalize
sequence, candidate verification through registry promotion; the
auto-release section becomes the deliberate patch sequence for v1.5.4;
and the Force-retagging section becomes a table of what is and is not
forced, since #2677's answer landed and almost nothing is any more.

The conceptual sections and mermaid diagrams under Release Candidates /
Regular Releases / Patch Releases are left alone -- cherry-pick onto the
line, PR against the line, is still the shape -- and main's docs/release.md
is not ported, because it documents promote-rc.yaml, changelog-rc.yaml and
retention.yaml, none of which are on this branch. Pointing at workflows
that are absent here would trade one wrong doc for another.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>