fix(migrations): pin KubeadmConfigTemplate on the 1.5 slot 45 migration and in the chart - #3892
Merged
Merged
Conversation
Migration slot 45 holds two different migrations on the two branches. On release-1.5 it is the SeaweedFS db-split repair (#3370 backport, which also bumped targetVersion to 46); on main and release-1.6 it is the helm.sh/resource-policy=keep pin on KubeadmConfigTemplate / KubeadmConfig that stops Helm pruning them while CAPI still references them from the kubeadm-backed MachineSet. v1.5.0 through v1.5.3 all ship targetVersion 45, so no released 1.5 cluster is stamped past 45 and every one of them still runs 1.6's slot 45 on the way up. Cutting v1.5.4 off release-1.5 as it stands is what first ships targetVersion 46: those clusters get stamped 46, a later 1.5.4 -> 1.6 upgrade runs `seq 46 53`, and 1.6's slot 45 never executes. The keep-pin is then silently skipped, the 1.6 kubernetes chart upgrade prunes KubeadmConfigTemplate while the live kubeadm-backed MachineSet still references it, and its bootstrap.configRef dangles through the Talos worker rollover. Tenant Kubernetes only; a noisy broken upgrade rather than data loss, but not self-healing. Fold the pin into release-1.5's slot 45 so that slot does both jobs. A v1.5.4 cluster stamped 46 then skips 1.6's slot 45 harmlessly, because the annotation is already on the objects. Renumbering release-1.5's repair to 1.6's slot 53 was the alternative and does not work: run-migrations.sh hard-fails on a missing slot file, so it would need no-op stubs for 46-52, and a cluster stamped 54 would run `seq 54 53` and skip every real 1.6 migration. The pin logic is ported into release-1.5's idiom rather than copied from main: this branch's slots are #!/bin/sh (busybox ash in the migrations image) and have no lib/cozystack-version.sh, so the helper is POSIX-clean with prefixed globals instead of `local`, and slot 45 keeps its existing inline labelled version stamp. The selector is a LABEL, app.kubernetes.io/managed-by=Helm, which Helm injects into everything it applies; meta.helm.sh/release-name is an annotation and would match nothing while reporting a clean run. Confirmed against a live v1.5 stand stamped 45: the chart's KubeadmConfigTemplate carries that label and no resource-policy, and the KubeadmConfig children CAPI spawns carry no managed-by at all, so they are correctly left alone. Fails closed but aggregates: every object that can be pinned is pinned before a non-zero return aborts ahead of the version stamp, so the Job retries the whole slot and reports all failures at once. The single per-object exception is an object that disappears between the fleet scan and the annotate — an app being deleted concurrently with the hook. Helm cannot prune what no longer exists, so that is a skip rather than a failed pre-upgrade hook blocking the upgrade. "not found" is accepted only there, never for the fleet scan, where a list never answers NotFound and accepting it would let a real failure read as an empty fleet. The SeaweedFS repair runs first and the pin second. Both fail closed ahead of the stamp and the Job retries the whole slot, so a half-completed attempt is safe either way; but on an attempt where only one half gets to run it should be the one whose failure is irreversible, and running the pin first would stop the SeaweedFS repair from being attempted at all on that pass. targetVersion stays 46 and no slot is added or renumbered. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Drives the pin end-to-end through the migration script, in the harness the SeaweedFS half already uses: the real slot 45 executed by path inside the migrations image's own pinned alpine base, so the interpreter under test is the busybox ash that run-migrations.sh actually gives it, with only the cluster boundary faked. The property asserted is the annotation landing on the object, never a function being callable. Thirteen cases: an unannotated Helm-managed KubeadmConfigTemplate gets pinned and the slot stamps 46; several templates are pinned each in its own namespace; an already-pinned one is skipped WITHOUT a write; zero matching objects is a clean run; a CAPI-spawned KubeadmConfig is left alone because it is not Helm-managed; a cluster with no kubeadm bootstrap provider stamps cleanly; an object that vanishes between the scan and the pin is skipped rather than fatal; a failing fleet scan and a failed pin both abort without stamping; a partial failure still pins the rest and then aborts; both halves run in one pass with the SeaweedFS hand-over first; a SeaweedFS failure aborts before the pin is attempted; and migration 43 does not pin, since the two libs must stay independently sourceable. The fake kubectl MODELS THE LABEL SELECTOR rather than ignoring it, which is what makes the characteristic form of this bug detectable. Helm injects app.kubernetes.io/managed-by=Helm as a LABEL; meta.helm.sh/release-name is an ANNOTATION and can never be a selector, so using it matches nothing, the pin silently applies to nothing, and the run reports pinned=0 failures=0 and exits 0. A fake that answered every selector identically could not tell that apart from a working pin. An ABSENT selector lists everything, exactly as kubectl does, so dropping the selector shows up as the pin stamping keep on Machine-owned objects instead of as a quiet no-op. Also rewrites the nine pre-existing SeaweedFS "this did not happen" assertions, which could not fail. POSIX and bash both exempt a !-negated pipeline from errexit — "the -e setting shall be ignored ... if the command's return value is being inverted with !" — so `! grep -q X f` runs, returns 1, and the test carries on to report success. Measured in this harness, not inferred: with the two halves of slot 45 deliberately swapped, a `! grep -q 'PIN '` assertion stayed green while the command log plainly contained the PIN line. The replacement, [ "$(grep -c X f)" -eq 0 ], puts the result inside `[`, whose non-zero status does trip errexit, and is the form already used elsewhere in the file. Their claims turn out to have been true all along, so nothing about the migrations changes; what changes is that they are now checked — with the hand-over mutated to re-annotate a Cluster that already carries keep, "idempotent: a Cluster already owned by <name>-db AND carrying keep is left alone" now goes red where before it passed. Every case was confirmed against a deliberately broken pin before being made to pass: annotation-as-selector, no selector at all, best-effort instead of fail-closed, an unconditional write instead of read-then-skip, a vanished object treated as fatal, the halves swapped, and the pin never called. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Platform migration 45 pins the KubeadmConfigTemplate objects that exist when a cluster runs it. That leaves a hole: a template created AFTERWARDS — a new tenant Kubernetes cluster, or a nodeGroup added while on 1.5.4 — is born without the annotation, and 1.6's own slot 45 never runs for that cluster because a v1.5.4 cluster is stamped 46 and runs `seq 46 53`. The 1.6 kubernetes chart upgrade then prunes the unpinned template while the kubeadm-backed MachineSet still references it from bootstrap.configRef, which is the bug the migration exists to prevent. Render the annotation in the chart so every template is born pinned. The migration then covers only the objects that predate it, and the two together leave no window. KEEP IS SAFE ON THE UNINSTALL PATH, which is the only behaviour it changes here. CAPI stamps an ownerReference to the Cluster on this object — verified on a live v1.5 stand, where KubeadmConfigTemplate, MachineDeployment and KubevirtMachineTemplate all carry ownerReferences: [Cluster/<release>] that the chart itself does not render — and the Cluster is Helm-managed with no keep of its own. So `helm uninstall` deletes the Cluster and Kubernetes garbage collection reclaims the template through that reference. keep suppresses Helm's own delete; it has no bearing on owner-driven GC, so uninstall does not leak. The name is deterministic, <release>-<nodeGroup>, not content-hashed the way KubevirtMachineTemplate is, so keep cannot accumulate a copy per upgrade. The one residue is a removed nodeGroup, whose template Helm no longer prunes: it is inert because nothing references it, it is re-adopted in place if that nodeGroup returns, and GC reclaims it with the Cluster. Not added to KubeadmConfig: this chart renders no bare KubeadmConfig, and the ones CAPI spawns from the template are owned by their Machine and carry no app.kubernetes.io/managed-by label, so Helm never manages or prunes them and the annotation would have nothing to act on. Confirmed on the same live stand. Nothing else in this chart sets helm.sh/resource-policy, so no existing conditional is being overridden. Tests assert the annotation on the rendered object for a single nodeGroup and for every group when two are configured — the second because the annotation has to sit inside the per-nodeGroup range, and a group added later is exactly the one that would otherwise be left prunable. Both were confirmed red before being made to pass: with the annotation removed from the template they fail on `unknown path metadata.annotations["helm.sh/resource-policy"]`, which is the object being born unpinned rather than a broken assertion. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…to fail
The four assertions in "neither chart classifies on mutable claim
timestamps or liveness" were written as `! grep -qF X "$f"`, which cannot
fail: POSIX and bash both exempt a !-negated pipeline from errexit — "the
-e setting shall be ignored ... if the command's return value is being
inverted with !" — so the command runs, returns 1, and the test carries on
reporting success. Those four were the entire body of the test, so it
asserted nothing whatsoever.
Converting them mechanically turns out to fail, and for a reason worth
keeping: all four names DO appear in both files, in the prose that
explains why each was rejected as a discriminator. `creationTimestamp` and
`readyReplicas` are discussed at length in the comment blocks that record
the decision. So a raw grep would fail on the documentation of the very
invariant the test enforces.
Assert against the template logic instead, with {{/* */}} comment blocks
stripped by a small awk helper. Every comment opener in both files starts
its own line, so dropping whole lines cannot take code with it. What the
test now forbids is a LIVE reference, which is what it always meant.
Proven in both directions: it passes with the explanatory prose in place,
and it fails when a real `readyReplicas` lookup is injected into live
template code in either file.
The awk helper is deliberately a one-liner. cozytest.sh rewrites any bare
`}` in column 0 into `return 0` + `}`, which would corrupt a multi-line
awk body.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migration slot 45 holds a different migration here than on main and release-1.6. On this branch it is the seaweedfs db-split repair, added after v1.5.3 by
33611dfe7(the #3370 backport), which in the same commit bumpedmigrations.targetVersionfrom 45 to 46. On main and 1.6 slot 45 is thehelm.sh/resource-policy=keeppin onKubeadmConfigTemplate, and the seaweedfs repair sits at slot 53 instead.Nothing is broken today. v1.5.0 through v1.5.3 all ship
targetVersion: 45, so no released cluster is stamped past it and every one of them still runs 1.6's slot 45 on the way up. Cutting v1.5.4 is what first ships 46. After that a 1.5.4 to 1.6 upgrade runsseq 46 53and never executes 1.6's slot 45, so helm prunesKubeadmConfigTemplatewhile the live kubeadm-backed MachineSet still references it,bootstrap.configRefdangles and controller-manager floods through the talos worker rollover. Tenant kubernetes only, noisy broken upgrade rather than data loss.This folds the keep-pin into release-1.5's slot 45 so it does both jobs, and puts the annotation on the rendered template in
packages/apps/kubernetes/templates/cluster.yamlso templates created after the upgrade are born pinned.targetVersionstays 46.Why uninstall does not leak
keepsuppresses only helm's own delete. CAPI stamps anownerReferenceto theClusteron this object, which the chart does not render (checked on a live v1.5 stand, andMachineDeploymentandKubevirtMachineTemplatecarry the same reference). TheClusteritself is helm-managed with nokeep, sohelm uninstalldeletes it and GC reclaims the template through that reference.Template name is deterministic rather than content-hashed, so
keepcannot accumulate a copy per upgrade. One residue is the template of a removed nodeGroup, which helm no longer prunes. It is inert, re-adopted in place if the group comes back, and GC takes it with theCluster.Tests
Mutation-proven rather than revert-tested. The one worth naming: swapping the selector to the
meta.helm.sh/release-nameannotation makes the migration exit 0 reportingpinned=0 failures=0, which looks like a clean run, and the test catches it because the annotation never reached the object.Also converts four pre-existing negated-grep assertions in
hack/seaweedfs-guard-parity.bats.! grep -q X fcannot fail, posix and bash both exempt a!-negated pipeline from errexit, so those were comments that look like assertions. All of them hold once they are real.