refactor(build): standardize image tagging to fix concurrent PR push conflicts - #2711

…rt (#2855)

## What this PR does

[#2711](#2711) moved PR
builds to unique `pr-<N>-<sha>` tags (`PUBLISH_FLOATING=0`) to end
concurrent-push 409 conflicts, but left `--cache-from` pointing at
`$(REGISTRY)/<img>:latest`. PR builds set `REGISTRY` to the per-CI
registry, which never publishes `:latest`, so every PR build's registry
cache lookup `404`s and rebuilds cold — slowing builds and, on a loaded
runner, pushing them into the 30-minute job timeout.

This adds a `CACHE_REGISTRY` knob (default
`ghcr.io/cozystack/cozystack`, where release builds *do* publish
`:latest`) and points every `--cache-from` at it. Push targets
(`$(REGISTRY)`) and the unique per-PR tags are unchanged, so #2711's
anti-conflict behaviour is preserved while PR builds regain a warm cache
from the last release.

- `hack/common-envs.mk`: new `CACHE_REGISTRY ?=
ghcr.io/cozystack/cozystack` with rationale.
- 31 package Makefiles: 37 `--cache-from
type=registry,ref=$(REGISTRY)/…` → `$(CACHE_REGISTRY)/…`.

Verified with `make -n image REGISTRY=iad.ocir.io/…`: push tag stays on
OCIR (`…/cozystack-controller:pr-<N>-<sha>`), cache-from now reads
`ghcr.io/cozystack/cozystack/cozystack-controller:latest`. Makefile-only
change — no generated artifacts affected.

> Note: cache freshness is bounded by the last release's `:latest`.
PR-to-PR freshness (a `main`-built `:buildcache` tag) is a possible
follow-up, intentionally out of scope here.

### Release note

```release-note
fix(build): PR CI image builds now read layer cache from ghcr.io `:latest` (published by releases) instead of the per-CI registry where `:latest` was never pushed after #2711, restoring warm-start caching for PR builds.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated Docker build configuration across multiple components to use a
dedicated cache registry for improved build layer reuse.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

PR cozystack#2711 switched PR builds to push only :pr-<N>-<sha> tags (PUBLISH_FLOATING=0) to stop concurrent-push 409 conflicts, but left --cache-from pointing at $(REGISTRY)/<img>:latest. PR builds set REGISTRY to the per-CI registry (iad.ocir.io) and never publish :latest there, so every PR build's registry cache lookup 404s and rebuilds cold.

Add a CACHE_REGISTRY knob (default ghcr.io/cozystack/cozystack, where release builds do publish :latest) and point every --cache-from at it. Push targets ($(REGISTRY)) and the unique per-PR tags are unchanged, so the cozystack#2711 anti-conflict behaviour is preserved while PR builds regain a warm cache from the last release.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

`--cache-to type=inline` only exports the final stage. 25 of 38 Dockerfiles are multistage, so the expensive `builder` stages (Go/Node compiles) were never cached even with a warm tag. Switch every image to a shared mode=max registry cache.

- New `cache-args` macro (hack/common-envs.mk) emits `--cache-from` always and `--cache-to … mode=max` only when WRITE_CACHE=1, so PR builds stay read-only and concurrent PRs never race on the cache ref (the 409 class #2711 fixed for tags). oci-mediatypes/image-manifest keep the cache manifest portable across registries.

- Cache ref is CACHE_REGISTRY/<img>:buildcache, co-located with the build registry (CACHE_REGISTRY now defaults to $(REGISTRY)).

- New build-main.yaml warms :buildcache on every push to main (WRITE_CACHE=1), serialized via concurrency.

All 31 package image recipes converted from the inline cache-from/cache-to pair to `$(call cache-args,…)`; ubuntu-container-disk keeps a per-k8s-version cache tag. Verified with `make -n image` across all packages in both PR (read-only) and main (mode=max write) contexts.

Part of #2937.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…3268)

## What this PR does

Completes the last step of #2937 — no workflow depends on the persistent
self-hosted runner anymore, so it can be decommissioned. PR builds, the
main cache warmer, nightly, promote-rc, and pull-requests-release
already run on the ephemeral pool; this PR moves the stragglers:

- **`tags.yaml` / `prepare-release`** moves to the large ephemeral shape
(same as the main cache warmer). Release builds now warm-start from the
shared `mode=max` build cache in OCIR (written by `build-main.yaml` on
every push to main) while still pushing images to GHCR. Cache manifests
are registry-portable, and a missing cache ref degrades harmlessly into
a cold build. Without this wiring, every tag build on an ephemeral
runner would start 100% cold — the persistent runner used to warm-start
from local buildkit state.
- **`tags.yaml` / `generate-changelog` and `update-website-docs`**,
**`backport.yaml`**, **`update-releasenotes.yaml`** move to
GitHub-hosted runners: these jobs only need git and the GitHub API, no
docker or repo toolchain.
- **`pull-requests.yaml`**: the `debug` label no longer reroutes jobs to
the self-hosted runner — that path would queue forever once the runner
is gone. The label still gates the SSH breakpoint on e2e failure, which
works from ephemeral runners (the breakpoint connects out to the
rendezvous server).
- **`flux-shard-operator`** was the last image built with the stale
inline/`:latest` cache pattern (always cold since #2711); it now uses
the shared `cache-args` registry cache like every other image.

Operational notes for the actual decommission:

- The breakpoint rendezvous server (`BREAKPOINT_ENDPOINT`) is separate
infrastructure — if it happens to live on the same host as the runner,
it needs a new home before the host is retired.
- `tags.yaml` now uses the `OCIR_USER`/`OCIR_TOKEN` secrets on tag
events (already used there by `nightly.yaml` and `release-e2e` before
it).

### Screenshots

Not applicable — CI-only change.

### Release note

```release-note
ci: all CI jobs now run on ephemeral or GitHub-hosted runners; release tag builds warm-start from the shared registry build cache
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated multiple GitHub Actions workflows to use more consistent
hosted runner environments for checks, PR validation, tagging,
backports, and release note updates.
* Improved tag/release build setup for ephemeral runners, including
optional build toolchain setup and container registry authentication.
* Enhanced image build caching for the shard operator to improve
packaging efficiency.
* **Bug Fixes**
* Removed label-driven runner switching in pull request checks to make
build and test execution more predictable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

… artifacts

Supersedes the interim guard in this branch`s first commit, which skipped the
overlay entirely for release-line PRs. Skipping fixed the wrong images but left
those PRs testing their line`s last release: for any package the PR did not
rebuild, the committed ref is the released digest, so a component changed by an
earlier backport was exercised as its pre-backport binary until the next rc.

The overlay now reads `cozystack-packages:<base branch>` instead of always
`:main`, and build-release.yaml publishes that artifact for every maintained
`release-<major>.<minor>` branch the way build-main.yaml does for main: images
tagged with the branch, and the whole packages tree pushed with each reference
digest-pinned to what the run just built. Each base branch therefore has its own
generation to overlay from, which is what the original bug was really about —
#3437 failed install because main`s cozystack-controller served an aggregated
OpenAPI release-1.6`s charts could not validate against.

Three deliberate choices:

  * The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The
    per-release and rc staging branches promote-rc.yaml and tags.yaml create
    (release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their
    images come from the tag build, and rebuilding them would be waste.
  * WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per
    image and build-main.yaml is deliberately its only, serialized writer so
    concurrent builds cannot race on the cache manifest (the 409 class #2711
    fixed for image tags). A line build can overlap a main build, so writing here
    would reintroduce that race. Line builds read the cache.
  * A missing artifact still degrades to committed refs, but on a release line it
    now emits a ::warning:: naming the branch. Silent degradation is
    indistinguishable from a working overlay, which is how a mis-specified branch
    filter would hide for a whole release cycle.

Cost: one `make build` per push to a maintained line, i.e. per merged backport.

hack/overlay-main-images_test.bats pins the artifact tag to the base branch,
rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s
branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main,
setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test.
13/13 green; actionlint and zizmor clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

… artifacts (#3471)

## What this PR does

The PR finalize job overlays image refs for every package a PR did not
rebuild, so e2e and the installer do not test last-release images for
everything outside the PR's build matrix. It could only ever read
`cozystack-packages:main`, which meant a **release-line PR was handed
main's binaries to run against its own line's charts**.

#3437 is the demonstration: a one-line change on
`release-1.6` deactivating an app failed install deterministically,
twice, with

```
helmrelease/backupstrategy-controller: Helm install failed …
error validating data: SchemaError(github.com/cozystack/cozystack/pkg/apis/core/v1alpha1.Option.spec):
unknown model in reference: "github.com~1cozystack~1cozystack~1pkg~1apis~1core~1v1alpha1.OptionSpec"
```

main's `cozystack-controller` served an aggregated OpenAPI that
`release-1.6`'s charts could not validate against. Nothing in that PR
was broken; the lane was. Both branches carry `option_types.go` and key
`OptionSpec` identically in the committed generated OpenAPI, so this is
a generation mismatch at runtime, not a codegen drift.

This PR fixes it by giving every base branch its own artifact to overlay
from, rather than by turning the overlay off.

### The change

1. **`pull-requests.yaml`** reads `cozystack-packages:${BASE_REF}`
(`github.base_ref`) instead of a hardcoded `:main`.
2. **`build-release.yaml`** (new) publishes that artifact for maintained
`release-<major>.<minor>` branches exactly as `build-main.yaml` does for
main: every image tagged with the branch, and the whole packages tree
pushed with each reference digest-pinned to what the run just built.

An earlier revision of this branch simply skipped the overlay for
non-main bases. That fixed the wrong-images problem but left
release-line PRs testing their line's *last release*: for any package
the PR did not rebuild, the committed ref is the released digest, so a
component changed by an earlier backport was still exercised as its
pre-backport binary until the next rc. Per-line artifacts remove that
gap too, which is why the guard was replaced rather than kept.

### Three deliberate choices

**The trigger matches line branches only** (`release-[0-9]+.[0-9]+`).
The per-release and rc staging branches `promote-rc.yaml` and
`tags.yaml` create — `release-1.6.1`, `release-1.6.0-rc.4` — must not
trigger a full rebuild; their images come from the tag build and
rebuilding them is waste.

**`WRITE_CACHE` stays `0`.** `CACHE_REGISTRY/<img>:buildcache` is a
single ref per image and `build-main.yaml` is deliberately its only,
serialized writer so concurrent builds cannot race on the cache manifest
— the 409 class #2711 fixed for image tags. A line build can overlap a
main build, so writing here would reintroduce that race. Line builds
read the cache.

**A missing artifact still degrades to committed refs, but says so.** On
a release line it emits a `::warning::` naming the branch. Silent
degradation is indistinguishable from a working overlay, which is how a
mis-specified branch filter could hide for a whole release cycle.

### Cost

One `make build` per push to a maintained line — in practice per merged
backport. That is the price of release-line PRs testing their line's tip
instead of its last release.

### Verification

`hack/overlay-main-images_test.bats` pins the artifact tag to the base
branch, rejects a hardcoded `:main` in either overlay step, and pins
`build-release.yaml`'s branch filter, image tag and `WRITE_CACHE: '0'`.
Mutation-checked: restoring `:main`, setting `WRITE_CACHE: '1'`, and
broadening the filter to `release-*` each fail a test. 13/13 green;
`actionlint` and `zizmor` clean.

Worth an explicit ack in review: the branch-filter pattern is the one
thing no local test can prove, since only GitHub evaluates it. If it
does not match, `build-release` never runs and the new `::warning::` is
what surfaces it on the next release-line PR.

### Backport

`release-1.6` needs this too — for `pull_request` events GitHub builds
the workflow from the merge ref, so a release-line PR only stops
receiving main's images once the change is on its base branch.
#3472 carried the interim guard and is closed in
favour of backporting this instead.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added a “Build release line” workflow that builds and publishes images
and the packages artifact for maintained `release-<major>.<minor>`
branches.

* **Bug Fixes**
* Updated PR workflow finalization to use base-branch–specific package
overlays, avoiding incorrect main-branch image references when targeting
release branches.
* Improved fallback behavior when the base-branch packages artifact is
unavailable.

* **Tests**
* Added workflow wiring tests to verify base-branch artifact usage and
that each maintained release line publishes its own packages artifact.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

The workflow_dispatch trigger added for the freeze takes any ref, and the
"Run workflow" ref selector preselects the default branch — which is
exactly where this workflow's own recovery advice, and cut-prerelease's
warning, send an operator. IMAGE_TAG is github.ref_name, so a dispatch
from main spends a two-hour `make build` republishing
cozystack-packages:main and every main image tag while build-main.yaml
may be writing the same ones: the 409 tag-collision class #2711 fixed.

Nothing is ever mislabeled, since the content really is that ref, so the
cost is a wasted runner rather than a wrong artifact. Guard it anyway: it
is five lines, and the mis-click is easy.

Anchored at both ends so the per-release staging branches (release-1.6.1,
release-1.6.0-rc.4) do not qualify — those are built by tags.yaml — and
gated on ref_type so a tag named release-1.6 cannot pass the pattern on
its own. Runs before the checkout and the OCIR login, so a refused
dispatch never touches the credentials. Fails rather than skips: a
skipped job reads in the run list like one that produced the artifact.

The push trigger was already filtered to release-X.Y; this closes the
dispatch path to the same set.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>