Skip to content

fix(kubernetes): give the worker-adoption guard a remedy an operator can run - #3903

Draft
myasnikovdaniil wants to merge 3 commits into
feat/kubernetes-nodes-phase2bfrom
daniil/3315-adoption-guard-remedy
Draft

fix(kubernetes): give the worker-adoption guard a remedy an operator can run#3903
myasnikovdaniil wants to merge 3 commits into
feat/kubernetes-nodes-phase2bfrom
daniil/3315-adoption-guard-remedy

Conversation

@myasnikovdaniil

Copy link
Copy Markdown
Contributor

Suggestion for #3315, not meant to be merged to main. Base is IvanHunters branch feat/kubernetes-nodes-phase2b so the diff shows only what i am asking for. Cherry-pick, take the diff, or close this and write it your own way.

Three commits.

fix(kubernetes) rewrites the remedy in the render guard's fail message. Old text says run the platform migration, which cannot help anyone who reaches that fail, migration 54 ends in bare stamp_cozystack_version 55 so it never re-runs on a cluster stamped past it. New text gives two remedies. Pinning by hand covers all four object kinds because pin_pool_objects pins the machinehealthcheck, the workloadmonitor and every content-hashed kubevirtmachinetemplate too, and a pruned kmt that an in-flight machineset still references is unrecoverable permanently. Stamping back names exactly 54 because run-migrations.sh iterates seq CURRENT TARGET-1, uses --type merge because a label-less apply strips platform.cozystack.io/no-delete that migration 42 exists to repair, and says out loud that the patch alone does nothing since the hook job renders only during a real helm upgrade of the platform release. The kmt pipeline needs xargs -r -I{}, kubectl wants every resource before the first KEY=VALUE pair.

feat(migrations) records the pools migration 54 pinned but did not adopt, into a cozystack-migration-54-unadopted ConfigMap. The job carries helm.sh/hook-delete-policy: before-hook-creation, so today the next upgrade deletes the only place those warnings were written. Hooked at the top of pin_pool_objects only so the delegating caller does not double-count, non-fatal on apply so a bookkeeping write can never stall a fleet-wide upgrade hook, temp file rather than a counter because the cluster sweep and the group loop are both pipeline subshells. One new bats case in hack/migration-54-adopt-path.bats, red phase checked three ways.

docs(migrations) only discloses the un-type-checked cluster-level half of child_values. Deliberately not adding the check, it would create a sixth skip branch and swap one once-only outcome for another.

Ran locally: helm unittest for packages/apps/kubernetes 17 suites 142 tests, migration-54-adopt-path.bats 19 cases, migration-54-adopt-values.bats, cozystack-version-stamp.bats, helm unittest for kubernetes-nodes, migrations target check. No go touched so build, vet and generate do not apply.

The guard refuses to render a control-plane-only cluster whose worker
MachineDeployments were never adopted and are not pinned, because Helm
would otherwise prune them and delete the tenant's worker VMs. Its
remedy text told the operator to "run the platform migration that adopts
worker pools into KubernetesNodes", which is a no-op on every cluster
that can reach this fail: migration 54 ends in a bare
stamp_cozystack_version 55 and version-gated migrations do not re-run.
The one operator who ever sees this message was told to do the one thing
that cannot help.

Replace that sentence with two executable remedies, certain one first.

(A) Pin by hand, then re-run the upgrade. All four object kinds, not
just the MachineDeployment: pin_pool_objects pins the MachineDeployment,
the MachineHealthCheck, the WorkloadMonitor and every content-hashed
KubevirtMachineTemplate for exactly this reason, and the list_kmts
comment records that a pruned KMT an in-flight MachineSet still
references is unrecoverable permanently, because the child chart
re-emits only KMTs already carrying its own release name. A message
naming one kind would be a new trap wearing a fix's clothes.

The KMT command uses `xargs -r -I{}`, not a bare `xargs`: kubectl's
GetResourcesAndPairs requires every resource to precede the first
KEY=VALUE pair, so appending names after `helm.sh/resource-policy=keep`
fails with "all resources must be specified before annotation changes".
The message says so inline, so the form does not get "simplified" back.

(B) Stamp cozystack-version back to 54, then trigger a platform upgrade.
54 exactly, not "below 54": run-migrations.sh iterates
`seq $CURRENT_VERSION $((TARGET_VERSION - 1))`, so 54 against target 55
re-runs only migration 54. `--type merge` is load-bearing -- a
label-less apply under the default field manager strips
platform.cozystack.io/no-delete, the breakage migration 42 exists to
repair. The stamp alone does nothing, since the hook Job is rendered
only during a real Helm upgrade of the platform release and drift
detection is off, so the message says the trigger is install-specific
and the operator's to choose rather than pretending to be a one-liner.

The namespace is now interpolated into every command via an explicit
argument index (%[3]s), which leaves the existing %q/%v arguments in
place. The guard's condition and every other sentence are unchanged, and
both refusal cases in tests/worker_adoption_guard_test.yaml match on the
"refusing to render cluster" prefix, so they are untouched.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Migration 54 deliberately skips hostile stored shapes -- an over-long
child release name, an RFC-1123-invalid nodeGroups key, a non-object
group value or nodeGroups map, a non-scalar scalar field -- by pinning
the pool's worker objects with helm.sh/resource-policy=keep and moving
on, so one tenant's mangled HelmRelease cannot deadlock the platform
pre-upgrade hook for every tenant.

Until now the only trace of that decision was a WARNING on the hook Job's
log, and the Job carries helm.sh/hook-delete-policy:
before-hook-creation, so the next platform upgrade deletes it. A pool
left pinned-but-unadopted was then invisible: prune-proof, unmanaged,
and named nowhere an operator could look afterwards.

pin_pool_objects now appends "<cluster>/<group>" to a temp file and the
end of the script turns a non-empty file into the ConfigMap
cozystack-migration-54-unadopted. A file rather than a shell counter
because pin_pool_objects is only ever reached from inside the
per-cluster and per-group pipeline subshells, whose variables die with
them, while the emit point sits outside both `done`s. Hooked on
pin_pool_objects only -- pin_cluster_pools delegates to it, so
instrumenting both would double-count -- and above its invalid-name
early return, so a slash-bearing key that cannot name any object is
recorded too.

The record is diagnostic, not platform state, so it deliberately does
NOT carry platform.cozystack.io/no-delete: an operator who has acted on
it may delete it. And its apply is deliberately non-fatal: under
set -euo pipefail a failed apply would exit before the stamp and re-run
the whole adoption for every tenant over a bookkeeping write, so it
warns instead. adopt_one's foreign-owner refusal stays out of the
record -- it pins one object while the rest of the pool still adopts, so
listing the pool as unadopted would mislead.

The new bats case drives the 53-char overflow branch and asserts the
ConfigMap was applied under its own name, parses as YAML, and lists
exactly that one pool. Its red phase was checked both ways: with the
record removed it fails, and with the record emitted but the pool half
of each line dropped it also fails.

The fake kubectl now discriminates an applied manifest on its own kind
instead of on the word appearing anywhere in it. The record's note text
names "HelmRelease", and the substring match sent it down the child-HR
branch, where it was rejected as an HR with an empty metadata.name --
found by the new case on its first run. It also splits the version stamp
from any other ConfigMap so the record cannot masquerade as a stamp, and
writes the record's body to FAKE_APPLIED_CM so a test can assert which
pools it lists.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
… child_values

The bad_scalar guard type-checks the fields child_values copies out of
the per-group nodeGroups value, so a stored object where a scalar belongs
routes into the warn + pin + skip contract instead of fabricating a child
HelmRelease that cannot validate. Nothing says that the cluster-level
half of the same construction gets no such check, and the next reader
hits the guard before the fallbacks.

Record it where they will look: storageClass and
nodeHealthCheck.maxUnhealthy / nodeStartupTimeout are unvalidated and
fire only when the per-group key is absent, while the version/talos pick
is unconditional and therefore reaches every pool. A non-scalar stored
there is copied through, the child HelmRelease lands, and it fails
KubernetesNodes schema validation naming the offending field in its own
status.

Deliberately left as a disclosure rather than a sixth skip branch. A
check here would convert a legible child-HR failure into "pinned,
unmanaged, warning in a Job log the next upgrade deletes", and since
migration 54 stamps unconditionally that silent outcome would be
once-only. Loud there beats silent here. Comment only, no behaviour
change.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@coderabbitai

coderabbitai Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 69cb588b-e348-4592-88f5-2dc34ca66f0f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant