Skip to content

fix(backupstrategy-controller): give chBackupClientImage a producer - #3405

Merged
myasnikovdaniil merged 2 commits into
mainfrom
fix/platform-migrations-pin
Jul 22, 2026
Merged

fix(backupstrategy-controller): give chBackupClientImage a producer#3405
myasnikovdaniil merged 2 commits into
mainfrom
fix/platform-migrations-pin

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

This is a workaround

backupstrategy-controller should not be running the platform migrations image. This PR does not fix that — it makes the existing reuse safe so the v1.6.0 promote can proceed, and documents in the tree exactly what is wrong and what should replace it. Treat it as a stopgap with a TODO, not as an endorsement of the design.

What is actually wrong

The Altinity strategy Pod drives clickhouse-backup's HTTP API. Its script uses exactly four binaries — curl, jq, sleep, date. For that it currently pulls platform-migrations:

Size 349 MB (a dedicated alpine + curl + jq + ca-certificates measures 23 MB)
Also contains kubectl, helm, git, cozyhr, the etcd-migrate binary, every migration script, vendored etcd CRDs
ENTRYPOINT run-migrations.sh

Two consequences worth naming. A tenant-adjacent backup Pod carries cluster tooling it has no use for. And because the image's own entrypoint is the migration runner, the command: override in the strategy template is load-bearing — drop that one line and the Pod runs platform migrations with kubectl and helm already on the PATH.

It also creates the coupling that caused the bug this PR fixes: bumping ETCD_OPERATOR_VERSION or editing any migration script re-digests the image, forcing the backup client's pin to move for reasons that have nothing to do with backups.

The release blocker

platform-migrations was pinned at two different digests in the committed tree:

Location Pin Producer
packages/core/platform/values.yaml .migrations.image v1.5.0@sha256:8bf61f17… packages/core/platform/Makefile
backupstrategy-controller/values.yaml .chBackupClientImage v1.4.0-rc.2@sha256:17390197… none

Promotion retags by digest: hack/promote-retag.sh copies every collected <repo>@<digest> to <repo>:<stable-version>. Two digests under one repository produce two copies competing for the same destination tag — the first wins, the second hits the write-once guard, and the promotion fails. A dry run against main emits exactly that pair:

▸ ghcr.io/cozystack/cozystack/platform-migrations  sha256:17390197…
  → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9
▸ ghcr.io/cozystack/cozystack/platform-migrations  sha256:8bf61f17…
  → docker://ghcr.io/cozystack/cozystack/platform-migrations:v9.9.9

v1.6.0 is the first release cut through promotion rather than a full rebuild. Every earlier release ran make build, which restamped both copies as a side effect and hid the missing producer.

What this PR does

packages/core/platform/Makefile now stamps both keys with the same ref, so they move in lockstep by construction rather than by instruction, and the committed value is aligned to the digest core/platform already carried. Moving the runner forward two minors is safe: it is used only as a curl + jq container, and the image built at v1.5.0 carries them (git show v1.5.0:…/migrations/Dockerfileapk add … jq ca-certificates bash curl).

What should replace it

A dedicated ch-backup-client image built by this package's own Makefile — measured at 23 MB, 15× smaller, with none of the cluster tooling and none of the entrypoint hazard. That also deletes the cross-package stamp this PR adds, because the pin would be produced by the package that owns it.

The objection that has kept this reuse in place — "a second tag release CI would have to re-digest each cut" — does not survive contact with the actual defect. Every other first-party image already pays that cost, automatically. What broke here was not an extra tag; it was a pin with no producer, which a package-owned image cannot reproduce.

I am deliberately not doing that here: it means a new image, a new build target and a new digest introduced during a release cut. It belongs in a follow-up.

The guard

hack/image-pin-consistency.bats asserts the general invariant — no repository pinned at more than one digest — driven through the real promotion selector so it tracks whatever set promotion actually acts on, rather than a reimplementation that could drift from it. Plus a specific lockstep assertion so a recurrence names its cause. This check could not have been added before now: it fails on the very duplicate this PR removes.

Testing

  • hack/image-pin-consistency.bats — both tests pass; verified they fail when the drifted pin is restored.
  • make bats-unit-tests green (294 tests, exit 0).
  • make -C packages/system/backupstrategy-controller test — 11 tests, 3 suites, green.
  • helm template … --set backupStorage.bucketNameOverride=test-bucket renders the strategy Pod with the new ref.
  • The Makefile's yq step was simulated directly to confirm the relative path and key path resolve and that comments survive the in-place edit.
  • Merged locally with fix(ci): cover every image-ref storage shape in promote, retag and mirror #3404 onto main and re-run: the expanded ref collector in that PR surfaces no additional duplicate pins (314 tests, exit 0).

Relationship to other work

Relates to #3143 — this is one instance of its unowned-pin class, the one its discussion flagged as having no producer. The four never-built packages there remain untouched.

Independent of #3404 (promote/retag/mirror storage-shape coverage); the two touch no common files and can merge in either order. Both are needed before the v1.6.0 promote is re-cut.

Release note

Fixed a release-blocking duplicate pin: the ClickHouse backup client image is now stamped from the same source as the platform migrations image it reuses, so both carry one digest. Previously they could drift, which made promotion attempt to retag two digests to the same stable tag and fail. Reusing the migrations image here remains a documented workaround pending a dedicated backup-client image.

Summary by CodeRabbit

  • Bug Fixes

    • Ensured the platform migrations image digest is pinned consistently between the platform and backup strategy components.
    • Updated the backup strategy component to reference the new pinned platform-migrations digest.
  • Tests

    • Added promotion/retag consistency checks to fail on conflicting source digests for the same stable tag.
    • Added drift detection to ensure platform-migrations image references match across consumer configuration files.
  • Documentation

    • Expanded in-chart comments and values documentation clarifying the coupling and workaround behavior.

platform-migrations was pinned at two different digests: v1.5.0 in
packages/core/platform/values.yaml, stamped by its Makefile, and
v1.4.0-rc.2 as .backupStrategyController.chBackupClientImage, stamped by
nothing at all. The second is the Altinity strategy Pod's curl+jq runner,
deliberately reusing the migrations image rather than shipping a second
one-binary tag that release CI would have to re-digest every cut. That
reuse only holds while the two move together, and the values.yaml comment
said so -- but a comment was the only thing enforcing it, so the copy with
no producer froze two minors back.

This is a release blocker, not just staleness. Promotion retags by digest,
copying every collected <repo>@<digest> to <repo>:<stable>. Two digests
under one repository produce two copies competing for one destination tag:
the first wins and the second hits the write-once guard in
hack/promote-retag.sh, which fails the promotion. A dry run against the
tree emitted exactly that pair. v1.6.0 is the first release cut through
promotion rather than a full rebuild, so it would have been the first to
hit it -- every earlier release ran `make build`, which restamped both
copies as a side effect and hid the missing producer.

packages/core/platform/Makefile now stamps both keys with the same ref, so
they are in lockstep by construction rather than by instruction, and the
committed value is aligned to the digest core/platform already carried.
The image is used only as a curl + jq + ca-certificates runner, so moving
it forward two minors changes no behaviour the strategy depends on; it
renders and the chart's helm-unittest suite passes.

hack/image-pin-consistency.bats adds the general guard -- no repository
pinned at more than one digest, driven through the real promotion selector
so it tracks whatever set promotion acts on -- plus a specific lockstep
assertion so a recurrence names its cause. The general check could not be
added before now: it fails on the very duplicate this commit removes.

Relates to #3143 (the unowned-pin class) and unblocks the v1.6.0 promote.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@dosubot dosubot Bot added area/build Issues or PRs related to image build infrastructure, multi-arch support area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug labels Jul 21, 2026
@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The change synchronizes the platform-migrations image digest across two consumers, documents its backupstrategy-controller workaround, and adds Bats checks for duplicate promotion destinations and mismatched image pins.

Changes

Image pin consistency

Layer / File(s) Summary
Synchronize migration image consumers
packages/core/platform/Makefile, packages/system/backupstrategy-controller/values.yaml, packages/system/backupstrategy-controller/templates/...
The image-migrations target stamps both consumer values, the backupstrategy-controller pin is updated to the v1.5.0 digest, and the template documents the shared-image workaround and command override.
Validate promotion image pins
hack/image-pin-consistency.bats
Bats tests detect duplicate digest destinations during promotion and enforce identical platform-migrations pins across both values files.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested labels: area/platform, area/testing

Suggested reviewers: kvaps

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title matches the main change: backupstrategy-controller's chBackupClientImage now has an upstream producer in the platform migrations workflow.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/platform-migrations-pin

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size/L This PR changes 100-499 lines, ignoring generated files area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review labels Jul 21, 2026
…s a workaround

Running the platform migrations image as the ClickHouse backup client is
not a design decision worth defending, and the in-tree comments read as
though it were. Reframe them.

The strategy script uses exactly curl, jq, sleep and date. For that it
pulls a 349 MB image carrying kubectl, helm, git, cozyhr, the
etcd-migrate binary and every migration script into a tenant-adjacent
Pod, and that image's ENTRYPOINT is run-migrations.sh -- so the
`command:` override in the strategy template is load-bearing rather than
incidental. A dedicated alpine + curl + jq + ca-certificates image
measures 23 MB.

The standing objection to a dedicated image -- "a second tag release CI
would have to re-digest each cut" -- does not survive contact with the
actual defect. Every other first-party image already pays that cost
automatically. What broke was a pin with NO producer, which an image
owned by its own package cannot reproduce; that fix would also delete
the cross-package stamp introduced here.

Comment-only. No rendered output changes.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

LGTM with non-blocking notes

Root cause, fix, and regression test all verified locally; the two caveats below are documentation/maintainability observations, not defects.

Claim mismatches

All load-bearing PR-body claims independently re-derived and confirmed OK:

  • "Two digests under one repo make promotion retag two digests to the same tag" — confirmed. env -u REGISTRY hack/promote-retag.sh v9.9.9 --dry-run on the PR tree emits exactly one platform-migrations copy (sha256:8bf61f17…, rc=0, no dupes). Restoring the drifted v1.4.0-rc.2 pin reproduces the duplicate.
  • "Moving the runner forward two minors is safe: used only as curl+jq" — confirmed. git show v1.4.0-rc.2:…/migrations/Dockerfile and v1.5.0:…/migrations/Dockerfile both carry apk add … jq ca-certificates bash curl and the same ENTRYPOINT ["/usr/bin/run-migrations.sh"]; the command: override in strategy-altinity-default.yaml:53 is present and load-bearing.
  • "Makefile stamps both keys, comments survive" — confirmed by simulating the two yq --inplace lines against a scratch copy: both keys stamp to the same ref, relative path ../../system/backupstrategy-controller/values.yaml resolves, comment count unchanged (111 → 111).
  • "both tests pass; fail when drift restored" — confirmed via hack/cozytest.sh hack/image-pin-consistency.bats (green), then mutation to v1.4.0-rc.2 → RED, then restore → green.
  • "helm-unittest passes" — confirmed, 11/11 in 3 suites.
  • release-note block present.

Caveats

  • Phase 5b (upgrade + fresh install): the only cluster-facing change is the chBackupClientImage digest bump (packages/system/backupstrategy-controller/values.yaml:38). The strategy Pod is ephemeral (restartPolicy: Never, created per-BackupJob), so there is no long-lived resource to adopt and no migration is required; upgraded clusters simply pick up the new digest on the next BackupJob. Not a #2502-class running-workload break. Image is first-party (ghcr.io/cozystack/cozystack/platform-migrations), same repo as the already-mirrored migrations image, so air-gapped pullability is unchanged. No CRD/RBAC/PVC/schema surface touched. Verified, no breakage.
  • Phase 5c (toggle corners): the touched template block is gated by {{- if $bucketName -}} (strategy-altinity-default.yaml:15). Rendered two corners — defaults (no bucketNameOverride) and --set backupStorage.bucketNameOverride=test-bucket; both render admittably and the changed image: field is unconditional within the gate. The PR adds no new toggle or dependsOn.
  • Phase 5d (test adequacy): the fix carries a non-vacuous regression test that reproduces the release blocker (mutation → RED confirmed), auto-discovered by the BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats)) glob (Makefile:162), so it is live CI coverage. It is correctly written for the cozytest.sh runner rather than real bats (no run/$status/hooks). Adequate.
  • chart_lint reported 2 missing_refs (.Values.registries.config, .Values.registries.mirrors) and 1 render_error (repository.yaml:8 — OCIRepository lookup returns nil offline) against packages/core/platform. These are pre-existing harness artifacts of rendering the platform chart offline without runtime values: this PR touches only packages/core/platform/Makefile (verified git diff = Makefile only), and repository.yaml/containerd-registry-secret.yaml/values.yaml are byte-identical to main. Not introduced by this PR.

Recommended follow-ups

  • The cross-package stamp is a deliberate coupling the author documents: packages/core/platform/Makefile now writes into a sibling package's values.yaml as a side effect of image-migrations. Building the platform image in isolation mutates backupstrategy-controller/values.yaml; the two pins can momentarily diverge if only one package is rebuilt, though the new guard catches it at CI time. The author's own TODO — a dedicated ch-backup-client image owned by the backup package, which deletes this cross-package stamp — is the correct terminal shape and is explicitly deferred to a follow-up. Non-blocking; this PR is a correctly-labelled stopgap that unblocks the v1.6.0 promote.

@myasnikovdaniil
myasnikovdaniil merged commit bbfe9f9 into main Jul 22, 2026
16 of 17 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the fix/platform-migrations-pin branch July 22, 2026 11:41
myasnikovdaniil added a commit that referenced this pull request Jul 22, 2026
Promoting this branch fails in two independent ways, both landing after
finalize has already created the write-once stable tag, published the
GitHub release and moved :latest.

chBackupClientImage had no producer and froze at v1.4.0-rc.2 while
.migrations.image advanced to v1.6.0-rc.4, leaving platform-migrations
pinned at two digests. promote-retag.sh copies every collected
<repo>@<digest> to <repo>:<stable>, so both compete for one destination
tag: the lower-sorting v1.4.0-rc.2 digest wins it and the second copy
trips the write-once guard, aborting the retag with five repositories
still untagged and the installer chart unpublished. Align it to the
digest this branch already carries for the migrations image. Both come
from the same tree, so the runner keeps the curl, jq and ca-certificates
the Altinity strategy Pod needs, and that Pod overrides the ENTRYPOINT
with its own command.

promote-rc.yaml on main now calls hack/promote-rewrite-tags.sh, which
does not exist here, so the dispatch would fail outright. Vendor it and
hack/lib/image-refs.sh verbatim from main (identical blobs). That also
fixes the rewrite's coverage: the old inline glob enumerated 155 files
and missed 10 rc-bearing ones, which is why the earlier promote left 11
references reading v1.6.0-rc.4. The shared enumeration covers all 168
and fails the promotion if any rc reference survives.

Both fixes are on main via #3405 and #3404; this carries them onto the
rc.4 staging tree so v1.6.0 ships the bytes rc.4 tested.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/build Issues or PRs related to image build infrastructure, multi-arch support area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants