feat(kubernetes): default gpu-operator to an OS-provided driver on Talos - #3585

No actionable comments were generated in the recent review. 🎉

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands.}

End-to-end run done, on hardware, and it holds. Reporting it as promised rather than leaving the claim on unit tests alone.

Setup: a fresh single-node tenant cluster, one node group with an L40S (nvidia.com/AD102GL_L40S), the open-kernel-modules schematic, machine.kernel.modules added by hand since #3571 is not released yet, and no DaemonSet of any kind. The values under test were applied through addons.gpuOperator.valuesOverride with content identical to this PR's default.

The last row matters because a Talos node whose extension services cannot start reboots at roughly 70 minutes of uptime while still reporting Ready; 93 minutes on a single boot rules that out. Note a one-time reboot right after the install is normal and is not that failure — the node object is younger than its CAPI Machine, which is expected.

The L40S is the point: everything behind this configuration was previously only exercised on Blackwell (GB202). It works unchanged on Ada, so the set is not GPU-generation-specific.

What this run does not establish, stated plainly:

• It exercised the values, not this PR's default. The cluster runs the released chart, so the set arrived through valuesOverride; that the chart emits it by default is covered by the unit tests here and nothing more. Testing the default itself needs a release carrying this PR.
• It says nothing about whether NVreg_NvLinkDisable=1 is needed. An L40S is PCIe and has no NVLink, so the run shows the parameter is harmless and that CUDA initialises with it. Its necessity rests on the SXM passthrough case in production, not on this cluster.
• CDI injection was confirmed by its effect, not by inspecting containerd. Worth recording for whoever debugs this next: with DEVICE_LIST_STRATEGY=cdi-annotations no CDI annotation appears on the Pod object — the device plugin returns it in the Allocate response and kubelet passes it to the CRI runtime. Its absence from kubectl get pod -o yaml is expected and is not evidence that CDI failed. The evidence is nvidia-smi working inside a pod whose image ships no driver.
• HAMi remains untested and uncovered, as the PR body says.

Taking this out of draft on the strength of the above.

Two findings from the review: one taken, one I disagree with.

Taken — the missing assertions. 48007bd pins NVIDIA_DRIVER_ROOT, DEVICE_ID_STRATEGY: uuid and CDI_ANNOTATION_PREFIX: cdi.k8s.io/ on the device plugin, plus the validator's own NVIDIA_DRIVER_ROOT alongside DRIVER_ROOT. The reasoning is right: CDI device names key on the UUID strategy, the plugin resolves the driver from NVIDIA_DRIVER_ROOT, and the annotation prefix has to be the one containerd reads, so a regression in any of the three breaks device resolution while the two original assertions still pass.

Not taken — emptying kernelModuleConfig.content. I think this would make things worse rather than remove staleness.

That content is only read when an operator sets both driver.enabled: true and kernelModuleConfig.create: true, which is to say when they have deliberately re-enabled the driver container. In that configuration NVreg_NvLinkDisable=1 is the correct value for Cozystack, not an accident: individual GPUs are passed into worker VMs without the NVSwitches, so the driver otherwise waits forever for an NVLink fabric that can never come up — Fabric State stays In Progress and every CUDA call fails with system not yet initialized. Emptying the default means an operator who re-enables the driver container silently loses the parameter and lands in exactly that failure, which is the class of silent breakage this PR and #3571 exist to remove.

There is also no contradiction to resolve. "No variant enables it any more" and "the content is right when it is enabled" are both true at once: create: false already makes it inert, and content is the whole payload, so an operator who wants different parameters overrides that key and replaces it wholesale rather than merging into it. Nothing is silently inherited.

The comment above it was updated in this PR to say precisely that nothing enables it by default and that it is kept for an operator who re-enables the driver container, who then also has to set create: true. If that reads as stale rather than as a documented escape hatch, the fix is in the wording, and I would rather sharpen the comment than remove a default that is correct whenever it is reachable.