Skip to content

fix(kubernetes): render the talos-reconcile Job for the default md0 group - #3535

Merged
myasnikovdaniil merged 1 commit into
mainfrom
fix/default-md0-talos-reconcile-job
Aug 4, 2026
Merged

fix(kubernetes): render the talos-reconcile Job for the default md0 group#3535
myasnikovdaniil merged 1 commit into
mainfrom
fix/default-md0-talos-reconcile-job

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Fixes #3504.

The chart has two loops over worker node groups and they read different sources. templates/cluster.yaml iterates the effective set through the kubernetes.nodeGroups helper, whose else-branch supplies the built-in md0 group when the user declares none — that is how #2936 made md0 removable without a Helm values merge re-adding it. templates/talos/talos-reconcile-job.yaml iterated the raw .Values.nodeGroups map instead, which is empty in exactly that case.

The consequence is a dangling reference. The chart cannot render the TalosConfigTemplate itself — it needs the apiserver Service ClusterIP, the Talos CA and the Kubernetes CA, none of which exist while Helm is executing the template — so cluster.yaml deliberately stopped rendering it and the Job became its only producer, while the MachineDeployment still names it in spec.template.spec.bootstrap.configRef. On a cluster with the default nodeGroups, MachineDeployment/<release>-md0 renders and zero Jobs render, so TalosConfigTemplate/<release>-md0 is never created and CAPI blocks every Machine that ever joins the group. The same Job also patches KamajiControlPlane.spec.network.certSANs with the live Service ClusterIP, so that is skipped too.

Nothing fails at install time, because the built-in group carries minReplicas: 0. The failure waits for the first scale-up, which is the documented path rather than a corner: values.yaml tells operators that enabling the ingress-nginx addon on a cluster with default nodeGroups means waiting for the cluster-autoscaler to bring md0 up in response to the controller Pods becoming Pending, and the autoscaler is deployed unconditionally whenever the tenant has an etcd DataStore.

The fix is to range over the helper so the Job set tracks the MachineDeployment set exactly. A cluster that declares its own groups renders byte-identically — which is why the pinned content-hash fixtures in tests/talos_templates_test.yaml are untouched — and md0 stays removable, because the fix defers to the helper rather than merging md0 in unconditionally. On upgrade it is additive: the Job appears where there was none, applies the TalosConfigTemplate the MachineDeployment already expects, and the content-hash name suffix means Helm creates a fresh Job rather than attempting to patch an immutable one.

How this survived

Every helm-unittest fixture and every e2e suite declares md0 explicitly. tests/nodegroups_default_test.yaml does render the empty-nodeGroups case, but lists only templates/cluster.yaml, so it never looked at a Job. The two OIDC chainsaw lanes do install with an empty map, but assert that the HelmRelease exists rather than that it becomes Ready, so a worker that never boots is invisible to them. Offline helm template cannot reach this path at all: cluster.yaml's instanceType validator needs a live lookup for u1.medium.

tests/talos_reconcile_nodegroups_test.yaml closes the gap and pins both halves of the helper contract — an empty map yields exactly one md0 Job carrying GROUP_NAME=md0, and a two-group map yields exactly two Jobs with no md0 among them, so a future fix that merges md0 in unconditionally fails here. Reverting the template change fails the first case on the document count, so it guards the behaviour rather than restating it.

Screenshots

Not a UI change.

Downstream repositories

Walked the trigger map in docs/agents/contributing.md against the diff: it is one Helm template loop and one new helm-unittest suite inside packages/apps/kubernetes. No package is added, renamed or removed; no values.yaml, values.schema.json, ApplicationDefinition, version enum or default changes; nothing under hack/, no namespace, variant, label, annotation or metric renamed. The behaviour now matches what values.yaml already documents, so no reference page drifts.

Release note

fix(kubernetes): a tenant Kubernetes cluster created without an explicit `nodeGroups` map now gets a talos-reconcile Job for the built-in `md0` node group. Previously that Job rendered only for explicitly declared groups, so the default `md0` MachineDeployment referenced a TalosConfigTemplate nothing ever created: every worker the cluster-autoscaler added to `md0` stayed stuck waiting for its bootstrap config, and the KamajiControlPlane certSANs patch the same Job performs never ran. Clusters that declare their own node groups were unaffected.

Summary by CodeRabbit

  • Bug Fixes

    • Ensured the built-in md0 node group receives its Talos reconciliation Job when no node groups are configured.
    • Prevented duplicate reconciliation Jobs when custom node groups are specified.
    • Preserved correct group-specific settings and control-plane configuration in generated Jobs.
  • Tests

    • Added regression coverage for default and custom node-group scenarios.

…roup

The Job in templates/talos/talos-reconcile-job.yaml is the only
producer of the TalosConfigTemplate that each worker
MachineDeployment names in spec.template.spec.bootstrap.configRef;
cluster.yaml renders the reference but deliberately not the object.
Its loop read .Values.nodeGroups directly, while the
MachineDeployment loop reads the effective set through the
kubernetes.nodeGroups helper.

For a cluster that supplies no nodeGroups the two disagree: the
helper's else-branch emits the built-in md0 group, so the md0
MachineDeployment renders, but the raw map is empty and no Job
renders at all. Every Machine the cluster-autoscaler adds to md0
then blocks indefinitely on a TalosConfigTemplate that nothing will
ever create, and the KamajiControlPlane spec.network.certSANs patch
the same Job performs is skipped too.

Range over the helper so the Job set tracks the MachineDeployment
set exactly. No change for a cluster that declares its own groups,
and md0 stays removable: the helper's if-branch keeps a
user-supplied map authoritative.

The gap survived because every helm-unittest fixture and e2e suite
declares md0 explicitly. tests/nodegroups_default_test.yaml does
render the empty-nodeGroups case but lists only
templates/cluster.yaml, so it never looked at the Job. The new
suite pins both halves of the helper contract and fails without
this fix.

Fixes #3504

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@myasnikovdaniil myasnikovdaniil added the backport Should change be backported on previous release label Aug 4, 2026
@myasnikovdaniil myasnikovdaniil added the backport Should change be backported on previous release label Aug 4, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@github-actions github-actions Bot added area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Aug 4, 2026
@dosubot dosubot Bot added the area/testing Issues or PRs related to testing (e2e, bats, unit tests) label Aug 4, 2026
@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cae39952-1504-420d-b7b5-059f2e398ab1

📥 Commits

Reviewing files that changed from the base of the PR and between 3f967ec and 4e9c5ce.

📒 Files selected for processing (2)
  • packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml
  • packages/apps/kubernetes/tests/talos_reconcile_nodegroups_test.yaml

📝 Walkthrough

Walkthrough

The Talos reconcile Job loop now uses effective node groups. Regression tests cover the built-in md0 group and explicit worker0 and worker1 groups.

Changes

Talos reconcile node-group rendering

Layer / File(s) Summary
Use effective node groups and validate rendered Jobs
packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml, packages/apps/kubernetes/tests/talos_reconcile_nodegroups_test.yaml
The template uses kubernetes.nodeGroups. Tests verify default md0 rendering, explicit group rendering, Job names, environment variables, and command arguments.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related issues

Possibly related PRs

Suggested reviewers: kvaps

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the fix to render the Talos reconcile Job for the default md0 group.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/default-md0-talos-reconcile-job

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The fix is at the root cause and I could not break it.

Business context: a tenant cluster created with the default empty nodeGroups renders the md0 MachineDeployment but no talos-reconcile Job, so the TalosConfigTemplate the deployment references is never created and every autoscaler-added worker blocks on bootstrap; this makes the Job loop read the same effective group set as the MachineDeployment loop.

What I checked beyond reading the diff. The new suite fails on the merge-base template exactly as described: case 1 dies on the document count, case 2 passes. Reverting the loop to the raw map and, separately, making the helper merge md0 unconditionally both turn the suite red, so it guards the behaviour rather than restating it. Full chart suite is green at 191 tests, and the untouched content-hash fixtures in talos_templates_test.yaml confirm the explicit-groups render is byte-identical, so no existing cluster's Job name rotates. After this change no raw .Values.nodeGroups iteration remains in the chart, and the sibling kubernetes-nodes chart is single-pool with no map loop, so this bug class does not exist there.

@myasnikovdaniil
myasnikovdaniil merged commit 2f8c953 into main Aug 4, 2026
33 of 36 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the fix/default-md0-talos-reconcile-job branch August 4, 2026 10:19
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

myasnikovdaniil added a commit that referenced this pull request Aug 4, 2026
…b for the default md0 group (#3536)

# Description
Backport of #3535 to `release-1.6`.
IvanHunters added a commit that referenced this pull request Aug 4, 2026
Bring the worker-pool split branch up to date with main (111 commits),
picking up the node-join and CI fixes it was missing: the 18m tenant
node-join deadline, guest-fsync-off on ephemeral CI disks (#3455),
kubevirt-csi client rate-limit raise (#3428), the Cilium ingress-IP
race guard (#3430), and etcd-operator v0.5.4.

Conflict resolution: main's #3535 (render the talos-reconcile Job for
the default md0 group) modified packages/apps/kubernetes/templates/talos/
talos-reconcile-job.yaml, which this branch deletes because the worker
split moves that Job into the kubernetes-nodes chart. Kept the parent
template deleted and dropped the parent-scoped test
talos_reconcile_nodegroups_test.yaml: the empty-nodeGroups md0-default
gap it guards cannot occur in the per-pool child chart, and migration 54
materialises the implicit md0 into an explicit child HelmRelease.

targetVersion resolves to 55 (migration 54 on top of main's 53).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kubernetes Issues or PRs related to the tenant Kubernetes app area/testing Issues or PRs related to testing (e2e, bats, unit tests) backport Should change be backported on previous release kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[kubernetes] Default md0 node group gets no talos-reconcile Job, so scaling it from zero stalls

2 participants