feat(kubernetes): let a node group log the guest serial console - #3637
Conversation
A worker VM that stalls during boot before Talos apid answers is invisible to every diagnostic the platform has: no Node registers, no certificate request arrives, and talosctl needs the apid that never came up. The guest serial console does show that window, but KubeVirt attaches its guest-console-log container only when the VMI asks for it, and the platform turns console logging off cluster-wide. Add an opt-in per-node-group logSerialConsole that sets the KubeVirt field, which takes precedence over the cluster-wide setting. The key renders nothing unless true -- including when set to false -- because the KubevirtMachineTemplate is content-hash named, so a byte emitted for a group that did not ask for it renames the template and rolls every live worker VM. The description carries the two costs the field name does not show. Changing the value rolls the group it is set on, so the stuck VM whose console was wanted is replaced along with the rest and the setting buys visibility into the next boot rather than the current one. And the cluster-wide default it overrides exists because that same container has been seen holding virt-launcher in PodInitializing, which would take a group down rather than explain it. Worker VMs are emitted by two charts. Both carry the field, and the golden-parity check gains a case that keeps their output identical. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe PR adds optional worker guest serial-console logging. It propagates configuration through API types and Helm schemas, renders KubeVirt fields, and collects bounded console diagnostics during e2e failures. ChangesWorker serial-console logging
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant Test as Chainsaw test
participant Script as run-kubernetes.sh
participant API as Kubernetes API
participant Pod as virt-launcher Pod
participant Talos as talosctl
Test->>Script: Handle node-join failure
Script->>API: Verify guest-console-log attachment
API-->>Script: Return matching Pod data
Script->>API: Capture bounded console logs
API->>Pod: Read guest-console-log
Pod-->>API: Return console output
Script->>Talos: Capture in-guest diagnostics
Script-->>Test: Write console and Talos artifacts
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
a97bc4a to
3c3d6dd
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (2)
packages/apps/kubernetes-nodes/tests/serial_console_log_test.yaml (1)
37-53: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winAssert the hash-stability contract directly.
These tests verify only that
logSerialConsoleis absent. They do not verify that unset andfalseproduce the same rendered bytes orKubevirtMachineTemplate.metadata.name. A whitespace-only change can still change the content hash and roll worker VMs while these assertions pass. Add a regression check for the rendered output or template name.This follows the PR objective that unset and false values preserve identical output.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/apps/kubernetes-nodes/tests/serial_console_log_test.yaml` around lines 37 - 53, Extend the default and explicitly disabled cases in the serial console Helm tests to assert the hash-stability contract directly: verify that unset and false render identical output or produce the same KubevirtMachineTemplate.metadata.name, while retaining the existing absence assertions.hack/e2e-chainsaw/_lib/run-kubernetes.sh (1)
709-717: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueQuote
${pods}in the truncation count.
podsalready holds one Pod name per line, sowc -lreturns the same total when the expansion is quoted. Quoting removes the pathname expansion that shellcheck reports at line 714 and keeps the shell lint gate clean.♻️ Proposed fix
printf 'capture stopped after %s Pods; %s matched in total\n' \ - "${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \ + "${max_pods}" "$(printf '%s\n' "${pods}" | wc -l | tr -d ' ')" \ >"${report_dir}/COLLECTION-TRUNCATED.txt"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 709 - 717, Update the truncation count in the loop around the pod collection to quote the `${pods}` expansion passed to `printf`, preserving its one-Pod-per-line content while preventing pathname expansion and satisfying shell lint.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh`:
- Around line 709-717: Update the truncation count in the loop around the pod
collection to quote the `${pods}` expansion passed to `printf`, preserving its
one-Pod-per-line content while preventing pathname expansion and satisfying
shell lint.
In `@packages/apps/kubernetes-nodes/tests/serial_console_log_test.yaml`:
- Around line 37-53: Extend the default and explicitly disabled cases in the
serial console Helm tests to assert the hash-stability contract directly: verify
that unset and false render identical output or produce the same
KubevirtMachineTemplate.metadata.name, while retaining the existing absence
assertions.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: dd41ac4b-2579-40c7-a953-f9b5323ca050
📒 Files selected for processing (19)
api/apps/v1alpha1/kubernetes/types.goapi/apps/v1alpha1/kubernetesnodes/types.gohack/e2e-chainsaw/_lib/run-kubernetes.shhack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yamlhack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yamlhack/run-kubernetes-serial-console_test.batspackages/apps/kubernetes-nodes/README.mdpackages/apps/kubernetes-nodes/templates/nodegroup.yamlpackages/apps/kubernetes-nodes/tests/render-parity.shpackages/apps/kubernetes-nodes/tests/serial_console_log_test.yamlpackages/apps/kubernetes-nodes/values.schema.jsonpackages/apps/kubernetes-nodes/values.yamlpackages/apps/kubernetes/README.mdpackages/apps/kubernetes/templates/cluster.yamlpackages/apps/kubernetes/tests/serial_console_log_test.yamlpackages/apps/kubernetes/values.schema.jsonpackages/apps/kubernetes/values.yamlpackages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yamlpackages/system/kubernetes-rd/cozyrds/kubernetes.yaml
There was a problem hiding this comment.
🧹 Nitpick comments (2)
hack/e2e-chainsaw/_lib/run-kubernetes.sh (2)
1774-1775: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDeclare
attach_rcaslocal.
run_kubernetes_testdeclares its other variables withlocal.attach_rcleaks into the global scope. The value is reset to 0 before each use, so behavior does not change today.♻️ Proposed change
- attach_rc=0 + local attach_rc=0 cozy_assert_guest_console_attached || attach_rc=$?🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 1774 - 1775, Declare attach_rc as a local variable in run_kubernetes_test before assigning it, matching the function’s existing local variable declarations and preventing it from leaking into global scope.
726-728: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueQuote
${pods}in the truncation count.Shellcheck flags line 727 (SC2086). Pod names cannot contain glob characters, so the current form is safe today. Quoting removes the warning and keeps the count identical, because command substitution already stripped the trailing newline.
♻️ Proposed change
printf 'capture stopped after %s Pods; %s matched in total\n' \ - "${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \ + "${max_pods}" "$(printf '%s\n' "${pods}" | wc -l | tr -d ' ')" \ >"${report_dir}/COLLECTION-TRUNCATED.txt"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 726 - 728, Quote the ${pods} expansion in the truncation-count command substitution within the COLLECTION-TRUNCATED report generation, preserving the existing wc -l and whitespace-trimming behavior while eliminating ShellCheck SC2086.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh`:
- Around line 1774-1775: Declare attach_rc as a local variable in
run_kubernetes_test before assigning it, matching the function’s existing local
variable declarations and preventing it from leaking into global scope.
- Around line 726-728: Quote the ${pods} expansion in the truncation-count
command substitution within the COLLECTION-TRUNCATED report generation,
preserving the existing wc -l and whitespace-trimming behavior while eliminating
ShellCheck SC2086.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 7c4fd258-8c2b-4501-a451-3c0bec65a710
📒 Files selected for processing (4)
hack/e2e-chainsaw/_lib/run-kubernetes.shhack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yamlhack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yamlhack/run-kubernetes-serial-console_test.bats
🚧 Files skipped from review as they are similar to previous changes (3)
- hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml
- hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml
- hack/run-kubernetes-serial-console_test.bats
3c3d6dd to
e95dee2
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
hack/run-kubernetes-serial-console_test.bats (1)
30-46: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueOptional: add a wedge-read failure case to the mock.
The
*initContainerStatuses*branch always returns 0, socozy_report_guest_console_wedgeis never exercised on its read-failure path. That path returns 0 and prints nothing, which means a broken read is silently reported as "no wedge". A dedicated variable, for examplekubectl_wedge_rc, would pin that behaviour.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/run-kubernetes-serial-console_test.bats` around lines 30 - 46, Extend the *initContainerStatuses* branch of the mock kubectl handler to honor a dedicated kubectl_wedge_rc return code before printing kubectl_wedge_rows. Default it to success so existing tests remain unchanged, while allowing tests to simulate wedge-read failures and verify cozy_report_guest_console_wedge’s failure behavior.hack/e2e-chainsaw/_lib/run-kubernetes.sh (1)
754-756: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueQuote the expansion in the total-Pod count.
${pods}is unquoted insideprintf, so the shell applies pathname expansion in addition to word splitting. A Pod name is unlikely to contain a glob character, but the count is what the truncation marker reports, so make it deterministic. Quoting also clears the SC2086 hint.♻️ Proposed fix
printf 'capture stopped after %s Pods; %s matched in total\n' \ - "${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \ + "${max_pods}" "$(printf '%s\n' "${pods}" | wc -l | tr -d ' ')" \ >"${report_dir}/COLLECTION-TRUNCATED.txt"The count stays the same, because command substitution already stripped the trailing newline from
pods.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 754 - 756, Update the total-Pod count in the truncation report within the collection logic to quote the pods expansion passed to printf, preventing pathname expansion while preserving the existing line count and report output.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/run-kubernetes-serial-console_test.bats`:
- Around line 685-688: Update the assertion in the loop around the grep command
to use a negated if condition, so absence of “1m45s” succeeds and its presence
emits the error and returns 1. Preserve the existing failure message and return
behavior.
---
Nitpick comments:
In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh`:
- Around line 754-756: Update the total-Pod count in the truncation report
within the collection logic to quote the pods expansion passed to printf,
preventing pathname expansion while preserving the existing line count and
report output.
In `@hack/run-kubernetes-serial-console_test.bats`:
- Around line 30-46: Extend the *initContainerStatuses* branch of the mock
kubectl handler to honor a dedicated kubectl_wedge_rc return code before
printing kubectl_wedge_rows. Default it to success so existing tests remain
unchanged, while allowing tests to simulate wedge-read failures and verify
cozy_report_guest_console_wedge’s failure behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 4fe767b2-283e-438c-8a03-e99c1d11a4bd
📒 Files selected for processing (2)
hack/e2e-chainsaw/_lib/run-kubernetes.shhack/run-kubernetes-serial-console_test.bats
| grep -q '1m45s' "$f" && { | ||
| echo "$f still carries the pre-Pod-state figure 1m45s" >&2 | ||
| return 1 | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win
This assertion inverts its own result and fails the test on the passing path.
grep -q '1m45s' "$f" && { ...; return 1; } is the last command in each loop iteration. When the file does not contain 1m45s, grep exits 1, the && list is not taken, and the list status is 1. That status becomes the exit status of the for loop, which is the last command in the test, so the test fails exactly when the assertion holds. Use a negated if instead.
🐛 Proposed fix
- grep -q '1m45s' "$f" && {
- echo "$f still carries the pre-Pod-state figure 1m45s" >&2
- return 1
- }
+ if grep -q '1m45s' "$f"; then
+ echo "$f still carries the pre-Pod-state figure 1m45s" >&2
+ return 1
+ fi📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| grep -q '1m45s' "$f" && { | |
| echo "$f still carries the pre-Pod-state figure 1m45s" >&2 | |
| return 1 | |
| } | |
| if grep -q '1m45s' "$f"; then | |
| echo "$f still carries the pre-Pod-state figure 1m45s" >&2 | |
| return 1 | |
| fi |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/run-kubernetes-serial-console_test.bats` around lines 685 - 688, Update
the assertion in the loop around the grep command to use a negated if condition,
so absence of “1m45s” succeeds and its presence emits the error and returns 1.
Preserve the existing failure message and return behavior.
e95dee2 to
f724cb9
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
hack/e2e-chainsaw/_lib/run-kubernetes.sh (1)
766-772: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueQuote
${pods}in the truncation count.Shellcheck flags line 769 (SC2086). Pod names cannot contain glob characters, so the current form works, but the quoted form gives the same count and removes the warning.
♻️ Proposed fix
printf 'capture stopped after %s Pods; %s matched in total\n' \ - "${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \ + "${max_pods}" "$(printf '%s\n' "${pods}" | wc -l | tr -d ' ')" \ >"${report_dir}/COLLECTION-TRUNCATED.txt"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 766 - 772, Update the truncation count in the pod-collection logic around the break condition to quote the `${pods}` expansion when passing it to `printf`, preserving the existing line count while resolving the shellcheck SC2086 warning.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh`:
- Around line 766-772: Update the truncation count in the pod-collection logic
around the break condition to quote the `${pods}` expansion when passing it to
`printf`, preserving the existing line count while resolving the shellcheck
SC2086 warning.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 167038ee-03de-425f-96af-6c183a68788f
📒 Files selected for processing (4)
hack/e2e-chainsaw/_lib/run-kubernetes.shhack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yamlhack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yamlhack/run-kubernetes-serial-console_test.bats
🚧 Files skipped from review as they are similar to previous changes (2)
- hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml
- hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml
f724cb9 to
5e784f9
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
hack/e2e-chainsaw/_lib/run-kubernetes.sh (1)
782-784: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueQuote
${pods}in the truncation count.
${pods}is already newline-separated, so word splitting is not needed here. Quoting keeps the count correct and clears SC2086.♻️ Proposed change
printf 'capture stopped after %s Pods; %s matched in total\n' \ - "${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \ + "${max_pods}" "$(printf '%s\n' "${pods}" | wc -l | tr -d ' ')" \ >"${report_dir}/COLLECTION-TRUNCATED.txt"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 782 - 784, Quote the pods expansion in the truncation-count command within the collection reporting block, while preserving the existing newline count and whitespace trimming behavior. Update the printf invocation that feeds wc so it uses the already newline-separated pods value without word splitting.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh`:
- Around line 782-784: Quote the pods expansion in the truncation-count command
within the collection reporting block, while preserving the existing newline
count and whitespace trimming behavior. Update the printf invocation that feeds
wc so it uses the already newline-separated pods value without word splitting.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 3316afdf-222a-4d3c-8c3a-365230681a26
📒 Files selected for processing (3)
hack/e2e-chainsaw/_lib/run-kubernetes.shhack/run-kubernetes-serial-console_test.batspackages/apps/kubernetes-nodes/tests/render-parity.sh
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/apps/kubernetes-nodes/tests/render-parity.sh
5e784f9 to
0e1c7c8
Compare
The tenant Kubernetes suites fail the node-Ready budget with zero Nodes registered, and the time is lost inside the guest between the worker VMI reaching Running and Talos requesting its apid certificate. Nothing collected today covers that window: the existing in-guest capture drives talosctl, so it needs the very apid the worker never reached, and on the management side there is neither a Node nor a certificate request to read. Turn on the guest console log for the suite's node group and collect each virt-launcher's guest-console-log container from the management cluster. Reads start at the beginning of the stream rather than tailing it, so a guest that repaints its console cannot push the boot output out of the window; every read is bounded, and the walk stops at a cap that records both counts, so the collector cannot become the unbounded term in a failure path that already runs several others under one deadline. A read that returns no console is not filed as a quiet guest. Three outcomes get three labels, because only one of them says anything about the guest: the read failed and returned nothing, the read succeeded and returned nothing, or it broke off part way. Each read's stderr is kept beside the capture rather than inside it, since kubectl's own "container is not valid for pod" would otherwise fill the file and a console that never started would read as one that had spoken. That case is the headline one here, so the Pod state and events that explain it are collected alongside, once for the whole selector. Enabling the console overrides a cluster-wide setting the platform holds for an open upstream defect, so the run is also an experiment, and its own failure mode is the one that hides best: the container wedges virt-launcher in Init, no worker boots, and the suite reports "fewer than 2 tenant nodes Ready" -- the same sentence the failure under study produces. Read as the known signature, the experiment's answer would be filed as the flake it was meant to explain. So the failure path names that case first, before any other diagnostic, and only when it is true. The passing path checks separately that the container attached at all, and reports the ratio it saw rather than a bare success, since the check is deliberately satisfied by any one Pod. It keeps three outcomes apart too: a read that did not answer, a selector that matched no Pod, and Pods that matched and carry nothing. It reads the Pod name alongside the container names so the last two cannot collapse, and both container lists so a future move of the container cannot fail a run over a diagnostic. It runs last, after everything the suite exists to prove, because a debugging aid must not preempt the assertions that cover the cluster. The capture runs before the talosctl one because it is the only of the two that can describe a worker which never reached apid, the shape this failure keeps taking. kubectl reports a warning and an empty result alike, on stderr with a zero status, so the exit status decides which artifact its output lands in. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
0e1c7c8 to
50c5e2c
Compare
What this PR does
Both tenant Kubernetes e2e suites keep missing the node-Ready budget with zero Nodes registered, and the time goes inside the guest, between the worker VMI reaching Running and Talos asking for its apid certificate. Nothing collected today reaches that window. The in-guest capture the harness already has drives
talosctl, so it needs the apid the worker never got to, and the management side has neither a Node nor a certificate request to read. Four attempts at this failure have been designed from outside the guest; the most recent raised the budget from 12m to 18m and the same fingerprint came back at the new deadline.KubeVirt can stream a VM's guest serial console into a
guest-console-logcontainer beside virt-launcher, readable withkubectl logsand needing nothing from inside the guest — no talosctl, no client certificate, no helper Pod, no reachable apid. That is the one property this failure class demands, so this turns it on for the suite's worker node group and collects it into the diagnostic bundle when node-join fails, ahead of thetalosctlcapture that cannot describe the same runs.Reads start at the beginning of the stream rather than tailing it: a guest that repaints its console would push the boot output out of any
--tailwindow, while--limit-bytestruncates from the far end instead. Every read is bounded, and the walk stops at a cap that records both counts, so the collector cannot become an unbounded term in a failure path that already runs several others under one deadline.A read that returns no console is not filed as a quiet guest. Three outcomes get three labels, because only one of them says anything about the guest: the read failed and returned nothing, the read succeeded and returned nothing, or it broke off part way. Each read's stderr is kept beside the capture rather than inside it — kubectl's own
container guest-console-log is not valid for podwould otherwise fill the file, and a console that never started would read as one that had spoken. That case is the headline one here, so the Pod state and events that explain it are collected alongside, once for the whole selector.The switch is a new opt-in
logSerialConsoleon a node group, off by default. It renders nothing unless set to true, including when set to false, because theKubevirtMachineTemplateis content-hash named and a byte emitted for a group that did not ask for it renames the template and rolls every live worker VM on upgrade. I verified that by rendering the base tree and this branch: absent and explicit-false produce a byte-identical template name on both charts. Worker VMs come from two charts,kubernetesandkubernetes-nodes; both carry the field, and the golden-parity check between them gains a case that keeps their output identical.The passing path checks separately that the container actually attached, and runs that check last, after everything the suite exists to prove — a debugging aid should not preempt the assertions that cover the cluster. Without it the suite would first learn the setting was inert in the one run that needed the console, when it can no longer be recovered.
Relates to #3513.
The risk this carries, stated up front
The platform turns serial console logging off cluster-wide in the KubeVirt CR. That setting is a mitigation added in February 2026 for kubevirt/kubevirt#15989, where the
guest-console-logcontainer kept virt-launcher from starting and left VMs inPodInitializing. A per-VMlogSerialConsoletakes precedence over it —isSerialConsoleLogEnabledreads the VMI field first and only then falls back to the cluster setting — which is what makes this work, and also makes this the first thing on the platform to go through that mitigation. The mitigation was written against KubeVirt v1.6.x; the platform now ships v1.8.4, and the upstream issue is still open with no reproduction reported either way on the newer version.So the first run of these suites is also the experiment, and it doubles as a revalidation of that February mitigation on the version actually shipping, which nobody has done. The two outcomes are distinguishable rather than ambiguous: kubevirt/kubevirt#15989 shows up as virt-launcher wedged in
PodInitializingwith the VMI never reaching Running, while the failure this instruments shows the VMI Running and zero certificate requests. A red run with the first signature is an answer, not noise — and the collector is built so that it says which one happened instead of reporting silence. The blast radius is small: the product field is absent by default and only the e2e node group sets it, so backing this out is one line.The revert criterion, stated before the run rather than after it. If worker virt-launcher Pods sit in
Initonguest-console-logand no Node registers, kubevirt/kubevirt#15989 still reproduces on v1.8.4: drop thelogSerialConsole: trueline from the e2e node group, keep the chart field, and the upstream issue gains the reproduction on a current version that its thread does not have. If the Pods reach Running and the console logs carry boot output, the mitigation is a candidate for removal platform-wide, which is its own change. Either way the run answers something that was open before it.Dropping
virtualMachineOptions.disableSerialConsoleLogoutright was the alternative. It is a smaller diff and needs no values key, but it exposes every VM on the platform to a race nobody has re-tested, instead of one node group in one test. That is its own revalidation exercise and belongs in its own change, on evidence this one produces.Verification
make unit-testspasses. The new bats suite runs green under bothbashand adash/bin/sh, which matters becausehack/cozytest.shis#!/bin/shand an earlier revision of this branch carried a bash-only construct that failed silently there. Every test in it is killed by a specific mutation of the code it covers. Both charts' helm-unittest suites pass, the golden-parity script reports byte-identical output across all four cases including the new one, andmake generateis idempotent at the repo root.Reviewed by Claude across several rounds and by an independent second reader; the review loop found and fixed real defects in this branch, including a green-path check that had become unable to report the one thing it existed for.
Screenshots
Not a UI change.
Downstream repositories
Release note
Summary by CodeRabbit
logSerialConsolesetting for Kubernetes worker node groups.