Skip to content

fix(kubernetes): default worker disk storageClass to the application storageClass - #3129

Merged
Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
fix/kubernetes-worker-disk-storageclass
Jul 2, 2026
Merged

fix(kubernetes): default worker disk storageClass to the application storageClass#3129
Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
fix/kubernetes-worker-disk-storageclass

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Worker node VMs in a tenant Kubernetes cluster set evictionStrategy: External, so they live-migrate. Live migration requires the worker system DataVolume to be ReadWriteMany, and linstor-csi only grants RWX on a DRBD-backed StorageClass. When a node group leaves storageClass empty, the worker DataVolume template previously omitted storageClassName and fell through to the management cluster's default StorageClass — which is operator-configured and not guaranteed to be DRBD-backed. On any cluster whose default StorageClass is not a replicated/DRBD class, the worker disk lands on a non-RWX class and live migration breaks.

This changes the empty-storageClass fallback from the management cluster default to the application-level storageClass (default replicated, a DRBD/LINSTOR class): {{- with .group.storageClass | default $.Values.storageClass }}. A per-node-group storageClass still wins when set; when both the node group and the application storageClass are empty, storageClassName is omitted so the cluster default still applies (explicit opt-out preserved).

Doc and codegen surfaces (types.go, values.yaml, values.schema.json, README.md, and the generated kubernetes-rd ResourceDefinition) are regenerated in lockstep with the new field description. Chart unit tests cover the full contract: fallback to the application class, a custom application class, per-node-group override winning, and the both-empty omit path.

Upgrade impact — one-time worker-pool roll. The KubevirtMachineTemplate name embeds a hash of the rendered worker template body, and the MachineDeployment references that name. On the shipped default (md0 leaves storageClass empty) the body changes from "no storageClassName" to storageClassName: replicated, so on upgrade the template is renamed and Cluster API performs a one-time rolling replacement of the worker pool — for every existing tenant cluster, including those whose default StorageClass already was replicated (the roll is driven by the body hash, not by whether the effective class differs). This is the same class of roll any worker-template change triggers and is unavoidable with this approach. Two test pins document and guard the hash so future body changes are deliberate.

Release note

fix(kubernetes): worker node disks now fall back to the application-level storageClass (a replicated/DRBD class) instead of the management-cluster default when a node group leaves storageClass empty, so live migration of worker VMs gets the ReadWriteMany volume it requires. Upgrading rolls each tenant cluster's worker pool once.

Summary by CodeRabbit

  • Bug Fixes

    • Improved worker disk storageClassName fallback for empty per-node-group values: it now reliably uses the app-level replicated/DRBD StorageClass (rather than relying on the management-cluster default).
    • Ensured KubeVirt data volume templates and rendering behavior are consistent across single- and multi-node-group setups.
  • Documentation

    • Refined storageClass/RWX/DRBD guidance across configuration docs and schema descriptions, including the intended non-immutable behavior (optional/undefaulted).
  • Tests

    • Updated template test expectations to cover the new fallback vs omission cases, including naming assertions for rendered templates.

@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8126dd18-8d12-4258-a5d5-045fd1edaacd

📥 Commits

Reviewing files that changed from the base of the PR and between c2cf503 and 22c96d8.

📒 Files selected for processing (8)
  • api/apps/v1alpha1/kubernetes/types.go
  • packages/apps/kubernetes/README.md
  • packages/apps/kubernetes/templates/cluster.yaml
  • packages/apps/kubernetes/tests/cluster_test.yaml
  • packages/apps/kubernetes/tests/values/common.yaml
  • packages/apps/kubernetes/values.schema.json
  • packages/apps/kubernetes/values.yaml
  • packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
✅ Files skipped from review due to trivial changes (5)
  • packages/apps/kubernetes/values.schema.json
  • api/apps/v1alpha1/kubernetes/types.go
  • packages/apps/kubernetes/values.yaml
  • packages/apps/kubernetes/tests/values/common.yaml
  • packages/apps/kubernetes/README.md
🚧 Files skipped from review as they are similar to previous changes (3)
  • packages/apps/kubernetes/templates/cluster.yaml
  • packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
  • packages/apps/kubernetes/tests/cluster_test.yaml

📝 Walkthrough

Walkthrough

Changes the nodeGroup.storageClass empty-value fallback from the management-cluster default StorageClass to the application-level replicated/DRBD StorageClass. The Helm template now falls back to $.Values.storageClass, and the API docs, schemas, README, values files, and tests are updated to describe and verify the new behavior.

Changes

nodeGroup storageClass fallback to app-level storageClass

Layer / File(s) Summary
Helm template fallback and contract docs
packages/apps/kubernetes/templates/cluster.yaml, api/apps/v1alpha1/kubernetes/types.go, packages/apps/kubernetes/values.schema.json, packages/apps/kubernetes/values.yaml, packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
The template now uses the chart-level storageClass fallback when node-group storageClass is empty, and the related schema/comment text now describes the application-level fallback, RWX/DRBD constraints, and non-immutable note.
Tests and usage docs
packages/apps/kubernetes/tests/cluster_test.yaml, packages/apps/kubernetes/tests/values/common.yaml, packages/apps/kubernetes/README.md
Tests now cover fallback, precedence, and omission cases; the README and test values comments are updated to match the new storageClass and diskSize wording.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • cozystack/cozystack#2639: Touches the same NodeGroup.storageClass contract area in api/apps/v1alpha1/kubernetes/types.go and related immutability validation behavior.

Suggested labels

area/storage

Suggested reviewers

  • kvaps
  • lllamnyp
  • androndo
  • IvanHunters
  • sircthulhu
  • myasnikovdaniil

Poem

🐇 Hop, hop, the storage path is bright,
DRBD-backed fallback glows just right.
When nodeGroups leave their class unnamed,
The chart-level path is now proclaimed.
Flop-free ears twitch at RWX cheer!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: worker disk storageClass now falls back to the application storageClass.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/kubernetes-worker-disk-storageclass

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Jun 29, 2026
@lexfrei
Aleksei Sviridkin (lexfrei) marked this pull request as ready for review June 29, 2026 13:23
@dosubot dosubot Bot added the kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API label Jun 29, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request updates the worker node storage class configuration to ensure that worker VMs receive a DRBD-backed StorageClass, which is required for live migration. By defaulting to the application-level storage class rather than the management cluster's default, the system ensures that volumes are properly configured for ReadWriteMany access. This change includes necessary updates to documentation, schema files, and unit tests to reflect the new behavior and ensure cluster stability during upgrades.

Highlights

  • StorageClass Fallback Logic: Updated the worker node storage class fallback mechanism to prioritize the application-level storage class instead of the management cluster default, ensuring compatibility with DRBD-backed volumes required for live migration.
  • Upgrade Impact: This change triggers a one-time rolling replacement of worker pools for existing clusters due to the change in the rendered worker template body hash.
  • Documentation and Testing: Updated documentation, schema definitions, and added comprehensive unit tests to verify the new fallback behavior and guard against unexpected template changes.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Kubernetes application package to fall back to the application-level storageClass (defaulting to replicated) when the individual nodeGroup's storageClass is left empty. This ensures that worker VM persistent disks are provisioned on a DRBD-backed StorageClass supporting ReadWriteMany (RWX) for live migration, preventing issues with linstor-csi. The changes include updates to the Helm templates, schema definitions, documentation, and comprehensive unit tests to verify the fallback logic and guard against unexpected worker-pool rolling upgrades. As there are no review comments, I have no additional feedback to provide.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/apps/kubernetes/values.yaml`:
- Line 85: Update the StorageClass field documentation in values.yaml to remove
the implication that the fallback is always DRBD-backed. The comment for the
storageClass setting should say that worker node disks fall back to the
application-level storageClass, and that this application-level value defaults
to replicated; keep the rest of the RWX/immutability guidance intact. Use the
storageClass field description itself as the target location for the wording
change.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 73e8cb68-4d1f-4ac9-b674-ef959b32c7e4

📥 Commits

Reviewing files that changed from the base of the PR and between 4db5652 and 8ed9378.

📒 Files selected for processing (8)
  • api/apps/v1alpha1/kubernetes/types.go
  • packages/apps/kubernetes/README.md
  • packages/apps/kubernetes/templates/cluster.yaml
  • packages/apps/kubernetes/tests/cluster_test.yaml
  • packages/apps/kubernetes/tests/values/common.yaml
  • packages/apps/kubernetes/values.schema.json
  • packages/apps/kubernetes/values.yaml
  • packages/system/kubernetes-rd/cozyrds/kubernetes.yaml

Comment thread packages/apps/kubernetes/values.yaml

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

⚠️ MERGE CONFLICTmergeable: CONFLICTING against main. Needs a rebase before this can land.

The fix is correct and well-tested. When a node group leaves storageClass empty, the worker DataVolume previously omitted storageClassName and fell through to the management cluster's default StorageClass, which is not guaranteed to be DRBD-backed. Since worker VMs use evictionStrategy: External, their disks must be RWX-capable, and linstor-csi only grants RWX on DRBD-backed classes. The one-liner {{- with .group.storageClass | default $.Values.storageClass }} fixes the root cause, and the both-empty opt-out path (omit storageClassName) is correctly preserved.

All documentation surfaces (types.go, values.yaml, values.schema.json, README.md, cozyrds/kubernetes.yaml) are updated in lockstep. Test coverage is thorough: six new/updated cases cover the fallback, custom app SC, group override wins, and both-empty omit path. The two hash-pin tests that document the upgrade blast radius are a nice touch.

Upgrade blast-radius note

On upgrade, CAPI will rename the KubevirtMachineTemplate for every existing tenant cluster whose md0 leaves storageClass empty — even on clusters where the management cluster default was already replicated, because the roll is driven by the body hash, not the effective StorageClass. This is safe (same class, one roll) and is clearly called out in the release note, which is the right place for it. The hash-pin tests document it in the test suite.

Blocking

  • Rebase onto current main to resolve the merge conflict.

Non-blocking suggestions

  • See inline comment on the hash-pin tests.

documentIndex: 4

###############################################
# worker-pool roll-on-upgrade blast radius #

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hash-pin tests are a great mechanism for forcing deliberate attention to worker-template body changes. One quality-of-life improvement: add a brief comment above this section showing how to recompute the expected hash values when the template body changes intentionally. Otherwise future contributors will see a failing CI with a cryptic name mismatch and have to trace back to the sha256sum|trunc 6 logic in cluster.yaml.

# To recompute after an intentional template body change:
#   helm template test-k8s packages/apps/kubernetes \
#     -f packages/apps/kubernetes/tests/values/common.yaml \
#     --set nodeGroups.md0.instanceType=u1.medium \
#     | yq e 'select(.kind == "KubevirtMachineTemplate") | .metadata.name' -
# The suffix after the last '-' is the hash to update below.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebased onto current main and added the recompute comment you asked for, above the hash-pin section in cluster_test.yaml. I kept your helm template … | yq 'select(.kind == "KubevirtMachineTemplate") | .metadata.name' recipe with one adjustment so it reproduces the pinned hashes exactly: it renders with tests/values/common.yaml and the test-k8s release name (the same inputs the pins use) and leaves instanceType="" rather than u1.medium — setting instanceType triggers a live-cluster lookup() and changes the rendered body, so it wouldn't match …-63ff81. The comment lists both commands (default body, and --set storageClass="" for the omitted-body pin); I verified both reproduce the pins. Head is 22c96d8.

@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the fix/kubernetes-worker-disk-storageclass branch from 8ed9378 to c2cf503 Compare June 30, 2026 10:38
…storageClass

Worker node VMs set evictionStrategy: External, so they live-migrate.
Live migration requires the system DataVolume on a ReadWriteMany volume,
and linstor-csi only grants RWX on a DRBD-backed StorageClass. When a node
group leaves storageClass empty the template previously omitted the field
and fell through to the management cluster default StorageClass, which may
be a non-DRBD class the CSI driver rejects for RWX — breaking migration.

Fall back to the application-level storageClass (the replicated/DRBD class)
when the node group storageClass is empty, instead of the cluster default.
When both are empty the field is still omitted, preserving the explicit
cluster-default opt-out.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict

LGTM

Correct, well-scoped fix: the empty-storageClass fallback for worker VM DataVolumes now targets the application-level storageClass (default replicated, a DRBD/LINSTOR class) instead of the management-cluster default, which fixes live migration on clusters whose infra default is not DRBD-backed; all four render cases are tested and pass, codegen surfaces are in lockstep, and the one-time worker-pool roll is correctly documented and guarded.

Verification performed

  • Template render: helm template with tests/values/common.yaml produces storageClassName: "replicated" on the shipped default (md0 empty), template name test-k8s-md0-63ff81 matching the pin; --set storageClass="" produces no storageClassName (name test-k8s-md0-dcdf58, matching the omit pin). MachineDeployment infrastructureRef points at the hashed name, so the body-hash roll-on-upgrade claim is mechanically accurate.
  • helm unittest: 140/140 pass, including the four new cases (fallback to app class, custom app class, per-node-group override wins, both-empty omit) and the two hash pins.
  • go vet ./kubernetes/... (module api/apps/v1alpha1): clean. The types.go change is a doc-comment only.
  • Codegen lockstep: the new nodeGroups[].storageClass description is byte-identical across types.go, values.yaml, values.schema.json, README.md (padding recomputed), and packages/system/kubernetes-rd/cozyrds/kubernetes.yaml. Schema is valid JSON, cozyrds valid YAML, working tree clean.
  • Semantics: .Values.storageClass already denotes an infra-cluster StorageClass name in existing code (templates/helmreleases/csi.yaml:34 uses it as infraStorageClass), so reusing it for the worker DataVolume (also provisioned on the infra cluster) is consistent, not a tenant-vs-infra class mix-up.

Upgrade & fresh-install impact

  • Upgrade (existing tenants): one-time worker-pool roll for every tenant cluster whose md0 leaves storageClass empty (the shipped default), driven by the template-body hash — accurate as stated, and true even for clusters already effectively on replicated. Not data-lossy: the disk-system DataVolume is the ephemeral OS/kubelet/containerd disk re-imaged from the Talos factory artifact; workload data lives in separate LINSTOR-replicated tenant PVCs with independent lifecycle. The roll is disruptive (rolling worker replacement) but standard for any worker-template change. No migration script required.
  • Fresh install: defaulting to replicated cannot reference a non-existent StorageClass. The iaas bundle that provides KubeVirt (and thus tenant worker VMs) is hard-gated to bundles.system.variant in {isp-full, isp-full-generic} (packages/core/platform/templates/bundles/iaas.yaml:1-2, fail otherwise), and both variants unconditionally install LINSTOR (packages/core/platform/templates/bundles/system.yaml:27 and :96), which provisions replicated. isp-hosted lacks LINSTOR but also cannot enable iaas, so it cannot run these VMs at all. Operators who deliberately want the infra default retain the both-empty opt-out (set app-level storageClass: "").

@lexfrei
Aleksei Sviridkin (lexfrei) merged commit 9e89ae8 into main Jul 2, 2026
74 of 75 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the fix/kubernetes-worker-disk-storageclass branch July 2, 2026 09:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants