fix(kubernetes): give the worker-adoption guard a remedy an operator can run - #3903
Draft
myasnikovdaniil wants to merge 3 commits into
Draft
fix(kubernetes): give the worker-adoption guard a remedy an operator can run#3903myasnikovdaniil wants to merge 3 commits into
myasnikovdaniil wants to merge 3 commits into
Conversation
The guard refuses to render a control-plane-only cluster whose worker
MachineDeployments were never adopted and are not pinned, because Helm
would otherwise prune them and delete the tenant's worker VMs. Its
remedy text told the operator to "run the platform migration that adopts
worker pools into KubernetesNodes", which is a no-op on every cluster
that can reach this fail: migration 54 ends in a bare
stamp_cozystack_version 55 and version-gated migrations do not re-run.
The one operator who ever sees this message was told to do the one thing
that cannot help.
Replace that sentence with two executable remedies, certain one first.
(A) Pin by hand, then re-run the upgrade. All four object kinds, not
just the MachineDeployment: pin_pool_objects pins the MachineDeployment,
the MachineHealthCheck, the WorkloadMonitor and every content-hashed
KubevirtMachineTemplate for exactly this reason, and the list_kmts
comment records that a pruned KMT an in-flight MachineSet still
references is unrecoverable permanently, because the child chart
re-emits only KMTs already carrying its own release name. A message
naming one kind would be a new trap wearing a fix's clothes.
The KMT command uses `xargs -r -I{}`, not a bare `xargs`: kubectl's
GetResourcesAndPairs requires every resource to precede the first
KEY=VALUE pair, so appending names after `helm.sh/resource-policy=keep`
fails with "all resources must be specified before annotation changes".
The message says so inline, so the form does not get "simplified" back.
(B) Stamp cozystack-version back to 54, then trigger a platform upgrade.
54 exactly, not "below 54": run-migrations.sh iterates
`seq $CURRENT_VERSION $((TARGET_VERSION - 1))`, so 54 against target 55
re-runs only migration 54. `--type merge` is load-bearing -- a
label-less apply under the default field manager strips
platform.cozystack.io/no-delete, the breakage migration 42 exists to
repair. The stamp alone does nothing, since the hook Job is rendered
only during a real Helm upgrade of the platform release and drift
detection is off, so the message says the trigger is install-specific
and the operator's to choose rather than pretending to be a one-liner.
The namespace is now interpolated into every command via an explicit
argument index (%[3]s), which leaves the existing %q/%v arguments in
place. The guard's condition and every other sentence are unchanged, and
both refusal cases in tests/worker_adoption_guard_test.yaml match on the
"refusing to render cluster" prefix, so they are untouched.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Migration 54 deliberately skips hostile stored shapes -- an over-long child release name, an RFC-1123-invalid nodeGroups key, a non-object group value or nodeGroups map, a non-scalar scalar field -- by pinning the pool's worker objects with helm.sh/resource-policy=keep and moving on, so one tenant's mangled HelmRelease cannot deadlock the platform pre-upgrade hook for every tenant. Until now the only trace of that decision was a WARNING on the hook Job's log, and the Job carries helm.sh/hook-delete-policy: before-hook-creation, so the next platform upgrade deletes it. A pool left pinned-but-unadopted was then invisible: prune-proof, unmanaged, and named nowhere an operator could look afterwards. pin_pool_objects now appends "<cluster>/<group>" to a temp file and the end of the script turns a non-empty file into the ConfigMap cozystack-migration-54-unadopted. A file rather than a shell counter because pin_pool_objects is only ever reached from inside the per-cluster and per-group pipeline subshells, whose variables die with them, while the emit point sits outside both `done`s. Hooked on pin_pool_objects only -- pin_cluster_pools delegates to it, so instrumenting both would double-count -- and above its invalid-name early return, so a slash-bearing key that cannot name any object is recorded too. The record is diagnostic, not platform state, so it deliberately does NOT carry platform.cozystack.io/no-delete: an operator who has acted on it may delete it. And its apply is deliberately non-fatal: under set -euo pipefail a failed apply would exit before the stamp and re-run the whole adoption for every tenant over a bookkeeping write, so it warns instead. adopt_one's foreign-owner refusal stays out of the record -- it pins one object while the rest of the pool still adopts, so listing the pool as unadopted would mislead. The new bats case drives the 53-char overflow branch and asserts the ConfigMap was applied under its own name, parses as YAML, and lists exactly that one pool. Its red phase was checked both ways: with the record removed it fails, and with the record emitted but the pool half of each line dropped it also fails. The fake kubectl now discriminates an applied manifest on its own kind instead of on the word appearing anywhere in it. The record's note text names "HelmRelease", and the substring match sent it down the child-HR branch, where it was rejected as an HR with an empty metadata.name -- found by the new case on its first run. It also splits the version stamp from any other ConfigMap so the record cannot masquerade as a stamp, and writes the record's body to FAKE_APPLIED_CM so a test can assert which pools it lists. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
… child_values The bad_scalar guard type-checks the fields child_values copies out of the per-group nodeGroups value, so a stored object where a scalar belongs routes into the warn + pin + skip contract instead of fabricating a child HelmRelease that cannot validate. Nothing says that the cluster-level half of the same construction gets no such check, and the next reader hits the guard before the fallbacks. Record it where they will look: storageClass and nodeHealthCheck.maxUnhealthy / nodeStartupTimeout are unvalidated and fire only when the per-group key is absent, while the version/talos pick is unconditional and therefore reaches every pool. A non-scalar stored there is copied through, the child HelmRelease lands, and it fails KubernetesNodes schema validation naming the offending field in its own status. Deliberately left as a disclosure rather than a sixth skip branch. A check here would convert a legible child-HR failure into "pinned, unmanaged, warning in a Job log the next upgrade deletes", and since migration 54 stamps unconditionally that silent outcome would be once-only. Loud there beats silent here. Comment only, no behaviour change. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Suggestion for #3315, not meant to be merged to main. Base is IvanHunters branch
feat/kubernetes-nodes-phase2bso the diff shows only what i am asking for. Cherry-pick, take the diff, or close this and write it your own way.Three commits.
fix(kubernetes)rewrites the remedy in the render guard'sfailmessage. Old text says run the platform migration, which cannot help anyone who reaches thatfail, migration 54 ends in barestamp_cozystack_version 55so it never re-runs on a cluster stamped past it. New text gives two remedies. Pinning by hand covers all four object kinds becausepin_pool_objectspins the machinehealthcheck, the workloadmonitor and every content-hashed kubevirtmachinetemplate too, and a pruned kmt that an in-flight machineset still references is unrecoverable permanently. Stamping back names exactly54becauserun-migrations.shiteratesseq CURRENT TARGET-1, uses--type mergebecause a label-less apply stripsplatform.cozystack.io/no-deletethat migration 42 exists to repair, and says out loud that the patch alone does nothing since the hook job renders only during a real helm upgrade of the platform release. The kmt pipeline needsxargs -r -I{}, kubectl wants every resource before the firstKEY=VALUEpair.feat(migrations)records the pools migration 54 pinned but did not adopt, into acozystack-migration-54-unadoptedConfigMap. The job carrieshelm.sh/hook-delete-policy: before-hook-creation, so today the next upgrade deletes the only place those warnings were written. Hooked at the top ofpin_pool_objectsonly so the delegating caller does not double-count, non-fatal on apply so a bookkeeping write can never stall a fleet-wide upgrade hook, temp file rather than a counter because the cluster sweep and the group loop are both pipeline subshells. One new bats case inhack/migration-54-adopt-path.bats, red phase checked three ways.docs(migrations)only discloses the un-type-checked cluster-level half ofchild_values. Deliberately not adding the check, it would create a sixth skip branch and swap one once-only outcome for another.Ran locally: helm unittest for
packages/apps/kubernetes17 suites 142 tests,migration-54-adopt-path.bats19 cases,migration-54-adopt-values.bats,cozystack-version-stamp.bats, helm unittest forkubernetes-nodes, migrations target check. No go touched so build, vet and generate do not apply.