Skip to content

fix(migrations): pin KubeadmConfigTemplate on the 1.5 slot 45 migration and in the chart - #3892

Merged
myasnikovdaniil merged 4 commits into
release-1.5from
fix/migration-45-kubeadm-pin
Aug 19, 2026
Merged

fix(migrations): pin KubeadmConfigTemplate on the 1.5 slot 45 migration and in the chart#3892
myasnikovdaniil merged 4 commits into
release-1.5from
fix/migration-45-kubeadm-pin

Conversation

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

Migration slot 45 holds a different migration here than on main and release-1.6. On this branch it is the seaweedfs db-split repair, added after v1.5.3 by 33611dfe7 (the #3370 backport), which in the same commit bumped migrations.targetVersion from 45 to 46. On main and 1.6 slot 45 is the helm.sh/resource-policy=keep pin on KubeadmConfigTemplate, and the seaweedfs repair sits at slot 53 instead.

Nothing is broken today. v1.5.0 through v1.5.3 all ship targetVersion: 45, so no released cluster is stamped past it and every one of them still runs 1.6's slot 45 on the way up. Cutting v1.5.4 is what first ships 46. After that a 1.5.4 to 1.6 upgrade runs seq 46 53 and never executes 1.6's slot 45, so helm prunes KubeadmConfigTemplate while the live kubeadm-backed MachineSet still references it, bootstrap.configRef dangles and controller-manager floods through the talos worker rollover. Tenant kubernetes only, noisy broken upgrade rather than data loss.

This folds the keep-pin into release-1.5's slot 45 so it does both jobs, and puts the annotation on the rendered template in packages/apps/kubernetes/templates/cluster.yaml so templates created after the upgrade are born pinned. targetVersion stays 46.

Why uninstall does not leak

keep suppresses only helm's own delete. CAPI stamps an ownerReference to the Cluster on this object, which the chart does not render (checked on a live v1.5 stand, and MachineDeployment and KubevirtMachineTemplate carry the same reference). The Cluster itself is helm-managed with no keep, so helm uninstall deletes it and GC reclaims the template through that reference.

Template name is deterministic rather than content-hashed, so keep cannot accumulate a copy per upgrade. One residue is the template of a removed nodeGroup, which helm no longer prunes. It is inert, re-adopted in place if the group comes back, and GC takes it with the Cluster.

Tests

Mutation-proven rather than revert-tested. The one worth naming: swapping the selector to the meta.helm.sh/release-name annotation makes the migration exit 0 reporting pinned=0 failures=0, which looks like a clean run, and the test catches it because the annotation never reached the object.

Also converts four pre-existing negated-grep assertions in hack/seaweedfs-guard-parity.bats. ! grep -q X f cannot fail, posix and bash both exempt a !-negated pipeline from errexit, so those were comments that look like assertions. All of them hold once they are real.

Migration slot 45 holds two different migrations on the two branches. On
release-1.5 it is the SeaweedFS db-split repair (#3370 backport, which
also bumped targetVersion to 46); on main and release-1.6 it is the
helm.sh/resource-policy=keep pin on KubeadmConfigTemplate / KubeadmConfig
that stops Helm pruning them while CAPI still references them from the
kubeadm-backed MachineSet.

v1.5.0 through v1.5.3 all ship targetVersion 45, so no released 1.5
cluster is stamped past 45 and every one of them still runs 1.6's slot 45
on the way up. Cutting v1.5.4 off release-1.5 as it stands is what first
ships targetVersion 46: those clusters get stamped 46, a later
1.5.4 -> 1.6 upgrade runs `seq 46 53`, and 1.6's slot 45 never executes.
The keep-pin is then silently skipped, the 1.6 kubernetes chart upgrade
prunes KubeadmConfigTemplate while the live kubeadm-backed MachineSet
still references it, and its bootstrap.configRef dangles through the
Talos worker rollover. Tenant Kubernetes only; a noisy broken upgrade
rather than data loss, but not self-healing.

Fold the pin into release-1.5's slot 45 so that slot does both jobs. A
v1.5.4 cluster stamped 46 then skips 1.6's slot 45 harmlessly, because
the annotation is already on the objects.

Renumbering release-1.5's repair to 1.6's slot 53 was the alternative and
does not work: run-migrations.sh hard-fails on a missing slot file, so it
would need no-op stubs for 46-52, and a cluster stamped 54 would run
`seq 54 53` and skip every real 1.6 migration.

The pin logic is ported into release-1.5's idiom rather than copied from
main: this branch's slots are #!/bin/sh (busybox ash in the migrations
image) and have no lib/cozystack-version.sh, so the helper is POSIX-clean
with prefixed globals instead of `local`, and slot 45 keeps its existing
inline labelled version stamp. The selector is a LABEL,
app.kubernetes.io/managed-by=Helm, which Helm injects into everything it
applies; meta.helm.sh/release-name is an annotation and would match
nothing while reporting a clean run. Confirmed against a live v1.5 stand
stamped 45: the chart's KubeadmConfigTemplate carries that label and no
resource-policy, and the KubeadmConfig children CAPI spawns carry no
managed-by at all, so they are correctly left alone.

Fails closed but aggregates: every object that can be pinned is pinned
before a non-zero return aborts ahead of the version stamp, so the Job
retries the whole slot and reports all failures at once. The single
per-object exception is an object that disappears between the fleet scan
and the annotate — an app being deleted concurrently with the hook. Helm
cannot prune what no longer exists, so that is a skip rather than a
failed pre-upgrade hook blocking the upgrade. "not found" is accepted
only there, never for the fleet scan, where a list never answers NotFound
and accepting it would let a real failure read as an empty fleet.

The SeaweedFS repair runs first and the pin second. Both fail closed
ahead of the stamp and the Job retries the whole slot, so a
half-completed attempt is safe either way; but on an attempt where only
one half gets to run it should be the one whose failure is irreversible,
and running the pin first would stop the SeaweedFS repair from being
attempted at all on that pass.

targetVersion stays 46 and no slot is added or renumbered.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Drives the pin end-to-end through the migration script, in the harness the
SeaweedFS half already uses: the real slot 45 executed by path inside the
migrations image's own pinned alpine base, so the interpreter under test
is the busybox ash that run-migrations.sh actually gives it, with only the
cluster boundary faked.

The property asserted is the annotation landing on the object, never a
function being callable. Thirteen cases: an unannotated Helm-managed
KubeadmConfigTemplate gets pinned and the slot stamps 46; several
templates are pinned each in its own namespace; an already-pinned one is
skipped WITHOUT a write; zero matching objects is a clean run; a
CAPI-spawned KubeadmConfig is left alone because it is not Helm-managed; a
cluster with no kubeadm bootstrap provider stamps cleanly; an object that
vanishes between the scan and the pin is skipped rather than fatal; a
failing fleet scan and a failed pin both abort without stamping; a partial
failure still pins the rest and then aborts; both halves run in one pass
with the SeaweedFS hand-over first; a SeaweedFS failure aborts before the
pin is attempted; and migration 43 does not pin, since the two libs must
stay independently sourceable.

The fake kubectl MODELS THE LABEL SELECTOR rather than ignoring it, which
is what makes the characteristic form of this bug detectable. Helm injects
app.kubernetes.io/managed-by=Helm as a LABEL; meta.helm.sh/release-name is
an ANNOTATION and can never be a selector, so using it matches nothing,
the pin silently applies to nothing, and the run reports pinned=0
failures=0 and exits 0. A fake that answered every selector identically
could not tell that apart from a working pin. An ABSENT selector lists
everything, exactly as kubectl does, so dropping the selector shows up as
the pin stamping keep on Machine-owned objects instead of as a quiet
no-op.

Also rewrites the nine pre-existing SeaweedFS "this did not happen"
assertions, which could not fail. POSIX and bash both exempt a !-negated
pipeline from errexit — "the -e setting shall be ignored ... if the
command's return value is being inverted with !" — so `! grep -q X f`
runs, returns 1, and the test carries on to report success. Measured in
this harness, not inferred: with the two halves of slot 45 deliberately
swapped, a `! grep -q 'PIN '` assertion stayed green while the command log
plainly contained the PIN line. The replacement,
[ "$(grep -c X f)" -eq 0 ], puts the result inside `[`, whose non-zero
status does trip errexit, and is the form already used elsewhere in the
file. Their claims turn out to have been true all along, so nothing about
the migrations changes; what changes is that they are now checked — with
the hand-over mutated to re-annotate a Cluster that already carries keep,
"idempotent: a Cluster already owned by <name>-db AND carrying keep is
left alone" now goes red where before it passed.

Every case was confirmed against a deliberately broken pin before being
made to pass: annotation-as-selector, no selector at all, best-effort
instead of fail-closed, an unconditional write instead of read-then-skip,
a vanished object treated as fatal, the halves swapped, and the pin never
called.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Platform migration 45 pins the KubeadmConfigTemplate objects that exist
when a cluster runs it. That leaves a hole: a template created AFTERWARDS
— a new tenant Kubernetes cluster, or a nodeGroup added while on 1.5.4 —
is born without the annotation, and 1.6's own slot 45 never runs for that
cluster because a v1.5.4 cluster is stamped 46 and runs `seq 46 53`. The
1.6 kubernetes chart upgrade then prunes the unpinned template while the
kubeadm-backed MachineSet still references it from bootstrap.configRef,
which is the bug the migration exists to prevent.

Render the annotation in the chart so every template is born pinned. The
migration then covers only the objects that predate it, and the two
together leave no window.

KEEP IS SAFE ON THE UNINSTALL PATH, which is the only behaviour it
changes here. CAPI stamps an ownerReference to the Cluster on this object
— verified on a live v1.5 stand, where KubeadmConfigTemplate,
MachineDeployment and KubevirtMachineTemplate all carry
ownerReferences: [Cluster/<release>] that the chart itself does not
render — and the Cluster is Helm-managed with no keep of its own. So
`helm uninstall` deletes the Cluster and Kubernetes garbage collection
reclaims the template through that reference. keep suppresses Helm's own
delete; it has no bearing on owner-driven GC, so uninstall does not leak.

The name is deterministic, <release>-<nodeGroup>, not content-hashed the
way KubevirtMachineTemplate is, so keep cannot accumulate a copy per
upgrade. The one residue is a removed nodeGroup, whose template Helm no
longer prunes: it is inert because nothing references it, it is re-adopted
in place if that nodeGroup returns, and GC reclaims it with the Cluster.

Not added to KubeadmConfig: this chart renders no bare KubeadmConfig, and
the ones CAPI spawns from the template are owned by their Machine and
carry no app.kubernetes.io/managed-by label, so Helm never manages or
prunes them and the annotation would have nothing to act on. Confirmed on
the same live stand.

Nothing else in this chart sets helm.sh/resource-policy, so no existing
conditional is being overridden.

Tests assert the annotation on the rendered object for a single nodeGroup
and for every group when two are configured — the second because the
annotation has to sit inside the per-nodeGroup range, and a group added
later is exactly the one that would otherwise be left prunable. Both were
confirmed red before being made to pass: with the annotation removed from
the template they fail on `unknown path
metadata.annotations["helm.sh/resource-policy"]`, which is the object
being born unpinned rather than a broken assertion.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…to fail

The four assertions in "neither chart classifies on mutable claim
timestamps or liveness" were written as `! grep -qF X "$f"`, which cannot
fail: POSIX and bash both exempt a !-negated pipeline from errexit — "the
-e setting shall be ignored ... if the command's return value is being
inverted with !" — so the command runs, returns 1, and the test carries on
reporting success. Those four were the entire body of the test, so it
asserted nothing whatsoever.

Converting them mechanically turns out to fail, and for a reason worth
keeping: all four names DO appear in both files, in the prose that
explains why each was rejected as a discriminator. `creationTimestamp` and
`readyReplicas` are discussed at length in the comment blocks that record
the decision. So a raw grep would fail on the documentation of the very
invariant the test enforces.

Assert against the template logic instead, with {{/* */}} comment blocks
stripped by a small awk helper. Every comment opener in both files starts
its own line, so dropping whole lines cannot take code with it. What the
test now forbids is a LIVE reference, which is what it always meant.

Proven in both directions: it passes with the explanatory prose in place,
and it fails when a real `readyReplicas` lookup is injected into live
template code in either file.

The awk helper is deliberately a one-liner. cozytest.sh rewrites any bare
`}` in column 0 into `return 0` + `}`, which would corrupt a multi-line
awk body.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: caba2808-d99e-46bd-908c-bb73484ae0ab

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/bug Categorizes issue or PR as related to a bug size/XL This PR changes 500-999 lines, ignoring generated files labels Aug 18, 2026
@myasnikovdaniil
myasnikovdaniil merged commit b0d8f3b into release-1.5 Aug 19, 2026
8 of 9 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the fix/migration-45-kubeadm-pin branch August 19, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/bug Categorizes issue or PR as related to a bug size/XL This PR changes 500-999 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant