fix(backupstrategy-controller): give chBackupClientImage a producer - #3405
Conversation
platform-migrations was pinned at two different digests: v1.5.0 in packages/core/platform/values.yaml, stamped by its Makefile, and v1.4.0-rc.2 as .backupStrategyController.chBackupClientImage, stamped by nothing at all. The second is the Altinity strategy Pod's curl+jq runner, deliberately reusing the migrations image rather than shipping a second one-binary tag that release CI would have to re-digest every cut. That reuse only holds while the two move together, and the values.yaml comment said so -- but a comment was the only thing enforcing it, so the copy with no producer froze two minors back. This is a release blocker, not just staleness. Promotion retags by digest, copying every collected <repo>@<digest> to <repo>:<stable>. Two digests under one repository produce two copies competing for one destination tag: the first wins and the second hits the write-once guard in hack/promote-retag.sh, which fails the promotion. A dry run against the tree emitted exactly that pair. v1.6.0 is the first release cut through promotion rather than a full rebuild, so it would have been the first to hit it -- every earlier release ran `make build`, which restamped both copies as a side effect and hid the missing producer. packages/core/platform/Makefile now stamps both keys with the same ref, so they are in lockstep by construction rather than by instruction, and the committed value is aligned to the digest core/platform already carried. The image is used only as a curl + jq + ca-certificates runner, so moving it forward two minors changes no behaviour the strategy depends on; it renders and the chart's helm-unittest suite passes. hack/image-pin-consistency.bats adds the general guard -- no repository pinned at more than one digest, driven through the real promotion selector so it tracks whatever set promotion acts on -- plus a specific lockstep assertion so a recurrence names its cause. The general check could not be added before now: it fails on the very duplicate this commit removes. Relates to #3143 (the unowned-pin class) and unblocks the v1.6.0 promote. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
📝 WalkthroughWalkthroughThe change synchronizes the ChangesImage pin consistency
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…s a workaround Running the platform migrations image as the ClickHouse backup client is not a design decision worth defending, and the in-tree comments read as though it were. Reframe them. The strategy script uses exactly curl, jq, sleep and date. For that it pulls a 349 MB image carrying kubectl, helm, git, cozyhr, the etcd-migrate binary and every migration script into a tenant-adjacent Pod, and that image's ENTRYPOINT is run-migrations.sh -- so the `command:` override in the strategy template is load-bearing rather than incidental. A dedicated alpine + curl + jq + ca-certificates image measures 23 MB. The standing objection to a dedicated image -- "a second tag release CI would have to re-digest each cut" -- does not survive contact with the actual defect. Every other first-party image already pays that cost automatically. What broke was a pin with NO producer, which an image owned by its own package cannot reproduce; that fix would also delete the cross-package stamp introduced here. Comment-only. No rendered output changes. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
IvanHunters
left a comment
There was a problem hiding this comment.
Verdict
LGTM with non-blocking notes
Root cause, fix, and regression test all verified locally; the two caveats below are documentation/maintainability observations, not defects.
Claim mismatches
All load-bearing PR-body claims independently re-derived and confirmed OK:
- "Two digests under one repo make promotion retag two digests to the same tag" — confirmed.
env -u REGISTRY hack/promote-retag.sh v9.9.9 --dry-runon the PR tree emits exactly oneplatform-migrationscopy (sha256:8bf61f17…, rc=0, no dupes). Restoring the driftedv1.4.0-rc.2pin reproduces the duplicate. - "Moving the runner forward two minors is safe: used only as curl+jq" — confirmed.
git show v1.4.0-rc.2:…/migrations/Dockerfileandv1.5.0:…/migrations/Dockerfileboth carryapk add … jq ca-certificates bash curland the sameENTRYPOINT ["/usr/bin/run-migrations.sh"]; thecommand:override instrategy-altinity-default.yaml:53is present and load-bearing. - "Makefile stamps both keys, comments survive" — confirmed by simulating the two
yq --inplacelines against a scratch copy: both keys stamp to the same ref, relative path../../system/backupstrategy-controller/values.yamlresolves, comment count unchanged (111 → 111). - "both tests pass; fail when drift restored" — confirmed via
hack/cozytest.sh hack/image-pin-consistency.bats(green), then mutation tov1.4.0-rc.2→ RED, then restore → green. - "helm-unittest passes" — confirmed, 11/11 in 3 suites.
- release-note block present.
Caveats
- Phase 5b (upgrade + fresh install): the only cluster-facing change is the
chBackupClientImagedigest bump (packages/system/backupstrategy-controller/values.yaml:38). The strategy Pod is ephemeral (restartPolicy: Never, created per-BackupJob), so there is no long-lived resource to adopt and no migration is required; upgraded clusters simply pick up the new digest on the next BackupJob. Not a #2502-class running-workload break. Image is first-party (ghcr.io/cozystack/cozystack/platform-migrations), same repo as the already-mirrored migrations image, so air-gapped pullability is unchanged. No CRD/RBAC/PVC/schema surface touched. Verified, no breakage. - Phase 5c (toggle corners): the touched template block is gated by
{{- if $bucketName -}}(strategy-altinity-default.yaml:15). Rendered two corners — defaults (nobucketNameOverride) and--set backupStorage.bucketNameOverride=test-bucket; both render admittably and the changedimage:field is unconditional within the gate. The PR adds no new toggle ordependsOn. - Phase 5d (test adequacy): the fix carries a non-vacuous regression test that reproduces the release blocker (mutation → RED confirmed), auto-discovered by the
BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))glob (Makefile:162), so it is live CI coverage. It is correctly written for thecozytest.shrunner rather than real bats (norun/$status/hooks). Adequate. - chart_lint reported 2
missing_refs(.Values.registries.config,.Values.registries.mirrors) and 1render_error(repository.yaml:8— OCIRepository lookup returns nil offline) againstpackages/core/platform. These are pre-existing harness artifacts of rendering the platform chart offline without runtime values: this PR touches onlypackages/core/platform/Makefile(verifiedgit diff= Makefile only), andrepository.yaml/containerd-registry-secret.yaml/values.yamlare byte-identical to main. Not introduced by this PR.
Recommended follow-ups
- The cross-package stamp is a deliberate coupling the author documents:
packages/core/platform/Makefilenow writes into a sibling package'svalues.yamlas a side effect ofimage-migrations. Building the platform image in isolation mutatesbackupstrategy-controller/values.yaml; the two pins can momentarily diverge if only one package is rebuilt, though the new guard catches it at CI time. The author's own TODO — a dedicatedch-backup-clientimage owned by the backup package, which deletes this cross-package stamp — is the correct terminal shape and is explicitly deferred to a follow-up. Non-blocking; this PR is a correctly-labelled stopgap that unblocks the v1.6.0 promote.
Promoting this branch fails in two independent ways, both landing after finalize has already created the write-once stable tag, published the GitHub release and moved :latest. chBackupClientImage had no producer and froze at v1.4.0-rc.2 while .migrations.image advanced to v1.6.0-rc.4, leaving platform-migrations pinned at two digests. promote-retag.sh copies every collected <repo>@<digest> to <repo>:<stable>, so both compete for one destination tag: the lower-sorting v1.4.0-rc.2 digest wins it and the second copy trips the write-once guard, aborting the retag with five repositories still untagged and the installer chart unpublished. Align it to the digest this branch already carries for the migrations image. Both come from the same tree, so the runner keeps the curl, jq and ca-certificates the Altinity strategy Pod needs, and that Pod overrides the ENTRYPOINT with its own command. promote-rc.yaml on main now calls hack/promote-rewrite-tags.sh, which does not exist here, so the dispatch would fail outright. Vendor it and hack/lib/image-refs.sh verbatim from main (identical blobs). That also fixes the rewrite's coverage: the old inline glob enumerated 155 files and missed 10 rc-bearing ones, which is why the earlier promote left 11 references reading v1.6.0-rc.4. The shared enumeration covers all 168 and fails the promotion if any rc reference survives. Both fixes are on main via #3405 and #3404; this carries them onto the rc.4 staging tree so v1.6.0 ships the bytes rc.4 tested. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
This is a workaround
backupstrategy-controllershould not be running the platform migrations image. This PR does not fix that — it makes the existing reuse safe so the v1.6.0 promote can proceed, and documents in the tree exactly what is wrong and what should replace it. Treat it as a stopgap with a TODO, not as an endorsement of the design.What is actually wrong
The Altinity strategy Pod drives clickhouse-backup's HTTP API. Its script uses exactly four binaries —
curl,jq,sleep,date. For that it currently pullsplatform-migrations:alpine + curl + jq + ca-certificatesmeasures 23 MB)kubectl,helm,git,cozyhr, theetcd-migratebinary, every migration script, vendored etcd CRDsENTRYPOINTrun-migrations.shTwo consequences worth naming. A tenant-adjacent backup Pod carries cluster tooling it has no use for. And because the image's own entrypoint is the migration runner, the
command:override in the strategy template is load-bearing — drop that one line and the Pod runs platform migrations withkubectlandhelmalready on the PATH.It also creates the coupling that caused the bug this PR fixes: bumping
ETCD_OPERATOR_VERSIONor editing any migration script re-digests the image, forcing the backup client's pin to move for reasons that have nothing to do with backups.The release blocker
platform-migrationswas pinned at two different digests in the committed tree:packages/core/platform/values.yaml.migrations.imagev1.5.0@sha256:8bf61f17…packages/core/platform/Makefilebackupstrategy-controller/values.yaml.chBackupClientImagev1.4.0-rc.2@sha256:17390197…Promotion retags by digest:
hack/promote-retag.shcopies every collected<repo>@<digest>to<repo>:<stable-version>. Two digests under one repository produce two copies competing for the same destination tag — the first wins, the second hits the write-once guard, and the promotion fails. A dry run againstmainemits exactly that pair:v1.6.0is the first release cut through promotion rather than a full rebuild. Every earlier release ranmake build, which restamped both copies as a side effect and hid the missing producer.What this PR does
packages/core/platform/Makefilenow stamps both keys with the same ref, so they move in lockstep by construction rather than by instruction, and the committed value is aligned to the digestcore/platformalready carried. Moving the runner forward two minors is safe: it is used only as a curl + jq container, and the image built atv1.5.0carries them (git show v1.5.0:…/migrations/Dockerfile→apk add … jq ca-certificates bash curl).What should replace it
A dedicated
ch-backup-clientimage built by this package's own Makefile — measured at 23 MB, 15× smaller, with none of the cluster tooling and none of the entrypoint hazard. That also deletes the cross-package stamp this PR adds, because the pin would be produced by the package that owns it.The objection that has kept this reuse in place — "a second tag release CI would have to re-digest each cut" — does not survive contact with the actual defect. Every other first-party image already pays that cost, automatically. What broke here was not an extra tag; it was a pin with no producer, which a package-owned image cannot reproduce.
I am deliberately not doing that here: it means a new image, a new build target and a new digest introduced during a release cut. It belongs in a follow-up.
The guard
hack/image-pin-consistency.batsasserts the general invariant — no repository pinned at more than one digest — driven through the real promotion selector so it tracks whatever set promotion actually acts on, rather than a reimplementation that could drift from it. Plus a specific lockstep assertion so a recurrence names its cause. This check could not have been added before now: it fails on the very duplicate this PR removes.Testing
hack/image-pin-consistency.bats— both tests pass; verified they fail when the drifted pin is restored.make bats-unit-testsgreen (294 tests, exit 0).make -C packages/system/backupstrategy-controller test— 11 tests, 3 suites, green.helm template … --set backupStorage.bucketNameOverride=test-bucketrenders the strategy Pod with the new ref.yqstep was simulated directly to confirm the relative path and key path resolve and that comments survive the in-place edit.mainand re-run: the expanded ref collector in that PR surfaces no additional duplicate pins (314 tests, exit 0).Relationship to other work
Relates to #3143 — this is one instance of its unowned-pin class, the one its discussion flagged as having no producer. The four never-built packages there remain untouched.
Independent of #3404 (promote/retag/mirror storage-shape coverage); the two touch no common files and can merge in either order. Both are needed before the
v1.6.0promote is re-cut.Release note
Summary by CodeRabbit
Bug Fixes
platform-migrationsdigest.Tests
platform-migrationsimage references match across consumer configuration files.Documentation