Skip to content

feat(kubernetes): let a node group log the guest serial console - #3637

Merged
Aleksei Sviridkin (lexfrei) merged 2 commits into
mainfrom
feat/e2e-capture-tenant-serial-console
Aug 8, 2026
Merged

feat(kubernetes): let a node group log the guest serial console#3637
Aleksei Sviridkin (lexfrei) merged 2 commits into
mainfrom
feat/e2e-capture-tenant-serial-console

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Both tenant Kubernetes e2e suites keep missing the node-Ready budget with zero Nodes registered, and the time goes inside the guest, between the worker VMI reaching Running and Talos asking for its apid certificate. Nothing collected today reaches that window. The in-guest capture the harness already has drives talosctl, so it needs the apid the worker never got to, and the management side has neither a Node nor a certificate request to read. Four attempts at this failure have been designed from outside the guest; the most recent raised the budget from 12m to 18m and the same fingerprint came back at the new deadline.

KubeVirt can stream a VM's guest serial console into a guest-console-log container beside virt-launcher, readable with kubectl logs and needing nothing from inside the guest — no talosctl, no client certificate, no helper Pod, no reachable apid. That is the one property this failure class demands, so this turns it on for the suite's worker node group and collects it into the diagnostic bundle when node-join fails, ahead of the talosctl capture that cannot describe the same runs.

Reads start at the beginning of the stream rather than tailing it: a guest that repaints its console would push the boot output out of any --tail window, while --limit-bytes truncates from the far end instead. Every read is bounded, and the walk stops at a cap that records both counts, so the collector cannot become an unbounded term in a failure path that already runs several others under one deadline.

A read that returns no console is not filed as a quiet guest. Three outcomes get three labels, because only one of them says anything about the guest: the read failed and returned nothing, the read succeeded and returned nothing, or it broke off part way. Each read's stderr is kept beside the capture rather than inside it — kubectl's own container guest-console-log is not valid for pod would otherwise fill the file, and a console that never started would read as one that had spoken. That case is the headline one here, so the Pod state and events that explain it are collected alongside, once for the whole selector.

The switch is a new opt-in logSerialConsole on a node group, off by default. It renders nothing unless set to true, including when set to false, because the KubevirtMachineTemplate is content-hash named and a byte emitted for a group that did not ask for it renames the template and rolls every live worker VM on upgrade. I verified that by rendering the base tree and this branch: absent and explicit-false produce a byte-identical template name on both charts. Worker VMs come from two charts, kubernetes and kubernetes-nodes; both carry the field, and the golden-parity check between them gains a case that keeps their output identical.

The passing path checks separately that the container actually attached, and runs that check last, after everything the suite exists to prove — a debugging aid should not preempt the assertions that cover the cluster. Without it the suite would first learn the setting was inert in the one run that needed the console, when it can no longer be recovered.

Relates to #3513.

The risk this carries, stated up front

The platform turns serial console logging off cluster-wide in the KubeVirt CR. That setting is a mitigation added in February 2026 for kubevirt/kubevirt#15989, where the guest-console-log container kept virt-launcher from starting and left VMs in PodInitializing. A per-VM logSerialConsole takes precedence over it — isSerialConsoleLogEnabled reads the VMI field first and only then falls back to the cluster setting — which is what makes this work, and also makes this the first thing on the platform to go through that mitigation. The mitigation was written against KubeVirt v1.6.x; the platform now ships v1.8.4, and the upstream issue is still open with no reproduction reported either way on the newer version.

So the first run of these suites is also the experiment, and it doubles as a revalidation of that February mitigation on the version actually shipping, which nobody has done. The two outcomes are distinguishable rather than ambiguous: kubevirt/kubevirt#15989 shows up as virt-launcher wedged in PodInitializing with the VMI never reaching Running, while the failure this instruments shows the VMI Running and zero certificate requests. A red run with the first signature is an answer, not noise — and the collector is built so that it says which one happened instead of reporting silence. The blast radius is small: the product field is absent by default and only the e2e node group sets it, so backing this out is one line.

The revert criterion, stated before the run rather than after it. If worker virt-launcher Pods sit in Init on guest-console-log and no Node registers, kubevirt/kubevirt#15989 still reproduces on v1.8.4: drop the logSerialConsole: true line from the e2e node group, keep the chart field, and the upstream issue gains the reproduction on a current version that its thread does not have. If the Pods reach Running and the console logs carry boot output, the mitigation is a candidate for removal platform-wide, which is its own change. Either way the run answers something that was open before it.

Dropping virtualMachineOptions.disableSerialConsoleLog outright was the alternative. It is a smaller diff and needs no values key, but it exposes every VM on the platform to a race nobody has re-tested, instead of one node group in one test. That is its own revalidation exercise and belongs in its own change, on evidence this one produces.

Verification

make unit-tests passes. The new bats suite runs green under both bash and a dash /bin/sh, which matters because hack/cozytest.sh is #!/bin/sh and an earlier revision of this branch carried a bash-only construct that failed silently there. Every test in it is killed by a specific mutation of the code it covers. Both charts' helm-unittest suites pass, the golden-parity script reports byte-identical output across all four cases including the new one, and make generate is idempotent at the repo root.

Reviewed by Claude across several rounds and by an independent second reader; the review loop found and fixed real defects in this branch, including a green-path check that had become unable to report the one thing it existed for.

Screenshots

Not a UI change.

Downstream repositories

Release note

feat(kubernetes): worker node groups accept `logSerialConsole`, which streams a worker VM's guest serial console into a `guest-console-log` container readable with `kubectl logs`. Off by default. It is the only way to observe a worker that stalls during boot before Talos apid answers, when no Node and no certificate request exist to diagnose from. Enabling it rolls the node group and overrides the platform's cluster-wide `disableSerialConsoleLog`.

Summary by CodeRabbit

  • New Features
    • Added an optional logSerialConsole setting for Kubernetes worker node groups.
    • Streams worker VM guest serial-console output to a dedicated log container when enabled.
    • Configuration is disabled by default and available through Kubernetes application settings.
  • Bug Fixes
    • Improved worker failure diagnostics with bounded serial-console output and pod-state information.
    • Validates that enabled worker pools attach the expected console-log container.
  • Documentation
    • Documented configuration behavior, rollout effects, defaults, and timeout considerations.
  • Tests
    • Added coverage for rendering, attachment validation, capture limits, and failure scenarios.

A worker VM that stalls during boot before Talos apid answers is
invisible to every diagnostic the platform has: no Node registers,
no certificate request arrives, and talosctl needs the apid that
never came up. The guest serial console does show that window, but
KubeVirt attaches its guest-console-log container only when the VMI
asks for it, and the platform turns console logging off cluster-wide.

Add an opt-in per-node-group logSerialConsole that sets the KubeVirt
field, which takes precedence over the cluster-wide setting. The key
renders nothing unless true -- including when set to false -- because
the KubevirtMachineTemplate is content-hash named, so a byte emitted
for a group that did not ask for it renames the template and rolls
every live worker VM.

The description carries the two costs the field name does not show.
Changing the value rolls the group it is set on, so the stuck VM
whose console was wanted is replaced along with the rest and the
setting buys visibility into the next boot rather than the current
one. And the cluster-wide default it overrides exists because that
same container has been seen holding virt-launcher in
PodInitializing, which would take a group down rather than explain
it.

Worker VMs are emitted by two charts. Both carry the field, and the
golden-parity check gains a case that keeps their output identical.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
@github-actions github-actions Bot added size/XXL This PR changes 1000+ lines, ignoring generated files area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/feature Categorizes issue or PR as related to a new feature labels Aug 8, 2026
@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR adds optional worker guest serial-console logging. It propagates configuration through API types and Helm schemas, renders KubeVirt fields, and collects bounded console diagnostics during e2e failures.

Changes

Worker serial-console logging

Layer / File(s) Summary
Configuration contracts
api/apps/..., packages/apps/..., packages/system/...
Adds optional logSerialConsole settings with false defaults to API types and chart schemas.
Template rendering and validation
packages/apps/kubernetes*/templates/*, packages/apps/kubernetes*/tests/*, packages/apps/kubernetes-nodes/tests/render-parity.sh
Renders logSerialConsole: true only when enabled and tests default, disabled, and enabled cases.
Console attachment and capture
hack/e2e-chainsaw/_lib/run-kubernetes.sh
Adds bounded guest-console capture, attachment checks, error artifacts, Pod-state diagnostics, and truncation markers.
e2e integration and regression coverage
hack/e2e-chainsaw/kubernetes-*/chainsaw-test.yaml, hack/run-kubernetes-serial-console_test.bats
Enables logging for tenant workers, collects console output before Talos diagnostics, verifies passing paths, documents timeout budgets, and covers the diagnostic flow with Bats tests.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related issues

  • cozystack/terraform-provider-cozystack#26 — The issue requests Terraform support for the same logSerialConsole configuration added by this PR.

Possibly related PRs

Suggested labels: area/testing

Suggested reviewers: myasnikovdaniil

Sequence Diagram(s)

sequenceDiagram
  participant Test as Chainsaw test
  participant Script as run-kubernetes.sh
  participant API as Kubernetes API
  participant Pod as virt-launcher Pod
  participant Talos as talosctl
  Test->>Script: Handle node-join failure
  Script->>API: Verify guest-console-log attachment
  API-->>Script: Return matching Pod data
  Script->>API: Capture bounded console logs
  API->>Pod: Read guest-console-log
  Pod-->>API: Return console output
  Script->>Talos: Capture in-guest diagnostics
  Script-->>Test: Write console and Talos artifacts
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 36.36% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the new opt-in node-group guest serial-console logging feature.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/e2e-capture-tenant-serial-console

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
packages/apps/kubernetes-nodes/tests/serial_console_log_test.yaml (1)

37-53: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Assert the hash-stability contract directly.

These tests verify only that logSerialConsole is absent. They do not verify that unset and false produce the same rendered bytes or KubevirtMachineTemplate.metadata.name. A whitespace-only change can still change the content hash and roll worker VMs while these assertions pass. Add a regression check for the rendered output or template name.

This follows the PR objective that unset and false values preserve identical output.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/apps/kubernetes-nodes/tests/serial_console_log_test.yaml` around
lines 37 - 53, Extend the default and explicitly disabled cases in the serial
console Helm tests to assert the hash-stability contract directly: verify that
unset and false render identical output or produce the same
KubevirtMachineTemplate.metadata.name, while retaining the existing absence
assertions.
hack/e2e-chainsaw/_lib/run-kubernetes.sh (1)

709-717: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Quote ${pods} in the truncation count.

pods already holds one Pod name per line, so wc -l returns the same total when the expansion is quoted. Quoting removes the pathname expansion that shellcheck reports at line 714 and keeps the shell lint gate clean.

♻️ Proposed fix
       printf 'capture stopped after %s Pods; %s matched in total\n' \
-        "${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \
+        "${max_pods}" "$(printf '%s\n' "${pods}" | wc -l | tr -d ' ')" \
         >"${report_dir}/COLLECTION-TRUNCATED.txt"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 709 - 717, Update the
truncation count in the loop around the pod collection to quote the `${pods}`
expansion passed to `printf`, preserving its one-Pod-per-line content while
preventing pathname expansion and satisfying shell lint.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh`:
- Around line 709-717: Update the truncation count in the loop around the pod
collection to quote the `${pods}` expansion passed to `printf`, preserving its
one-Pod-per-line content while preventing pathname expansion and satisfying
shell lint.

In `@packages/apps/kubernetes-nodes/tests/serial_console_log_test.yaml`:
- Around line 37-53: Extend the default and explicitly disabled cases in the
serial console Helm tests to assert the hash-stability contract directly: verify
that unset and false render identical output or produce the same
KubevirtMachineTemplate.metadata.name, while retaining the existing absence
assertions.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: dd41ac4b-2579-40c7-a953-f9b5323ca050

📥 Commits

Reviewing files that changed from the base of the PR and between 605030b and a97bc4a.

📒 Files selected for processing (19)
  • api/apps/v1alpha1/kubernetes/types.go
  • api/apps/v1alpha1/kubernetesnodes/types.go
  • hack/e2e-chainsaw/_lib/run-kubernetes.sh
  • hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml
  • hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml
  • hack/run-kubernetes-serial-console_test.bats
  • packages/apps/kubernetes-nodes/README.md
  • packages/apps/kubernetes-nodes/templates/nodegroup.yaml
  • packages/apps/kubernetes-nodes/tests/render-parity.sh
  • packages/apps/kubernetes-nodes/tests/serial_console_log_test.yaml
  • packages/apps/kubernetes-nodes/values.schema.json
  • packages/apps/kubernetes-nodes/values.yaml
  • packages/apps/kubernetes/README.md
  • packages/apps/kubernetes/templates/cluster.yaml
  • packages/apps/kubernetes/tests/serial_console_log_test.yaml
  • packages/apps/kubernetes/values.schema.json
  • packages/apps/kubernetes/values.yaml
  • packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yaml
  • packages/system/kubernetes-rd/cozyrds/kubernetes.yaml

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
hack/e2e-chainsaw/_lib/run-kubernetes.sh (2)

1774-1775: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Declare attach_rc as local.

run_kubernetes_test declares its other variables with local. attach_rc leaks into the global scope. The value is reset to 0 before each use, so behavior does not change today.

♻️ Proposed change
-  attach_rc=0
+  local attach_rc=0
   cozy_assert_guest_console_attached || attach_rc=$?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 1774 - 1775, Declare
attach_rc as a local variable in run_kubernetes_test before assigning it,
matching the function’s existing local variable declarations and preventing it
from leaking into global scope.

726-728: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Quote ${pods} in the truncation count.

Shellcheck flags line 727 (SC2086). Pod names cannot contain glob characters, so the current form is safe today. Quoting removes the warning and keeps the count identical, because command substitution already stripped the trailing newline.

♻️ Proposed change
       printf 'capture stopped after %s Pods; %s matched in total\n' \
-        "${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \
+        "${max_pods}" "$(printf '%s\n' "${pods}" | wc -l | tr -d ' ')" \
         >"${report_dir}/COLLECTION-TRUNCATED.txt"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 726 - 728, Quote the
${pods} expansion in the truncation-count command substitution within the
COLLECTION-TRUNCATED report generation, preserving the existing wc -l and
whitespace-trimming behavior while eliminating ShellCheck SC2086.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh`:
- Around line 1774-1775: Declare attach_rc as a local variable in
run_kubernetes_test before assigning it, matching the function’s existing local
variable declarations and preventing it from leaking into global scope.
- Around line 726-728: Quote the ${pods} expansion in the truncation-count
command substitution within the COLLECTION-TRUNCATED report generation,
preserving the existing wc -l and whitespace-trimming behavior while eliminating
ShellCheck SC2086.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7c4fd258-8c2b-4501-a451-3c0bec65a710

📥 Commits

Reviewing files that changed from the base of the PR and between a97bc4a and 3c3d6dd.

📒 Files selected for processing (4)
  • hack/e2e-chainsaw/_lib/run-kubernetes.sh
  • hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml
  • hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml
  • hack/run-kubernetes-serial-console_test.bats
🚧 Files skipped from review as they are similar to previous changes (3)
  • hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml
  • hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml
  • hack/run-kubernetes-serial-console_test.bats

@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the feat/e2e-capture-tenant-serial-console branch from 3c3d6dd to e95dee2 Compare August 8, 2026 01:09

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
hack/run-kubernetes-serial-console_test.bats (1)

30-46: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Optional: add a wedge-read failure case to the mock.

The *initContainerStatuses* branch always returns 0, so cozy_report_guest_console_wedge is never exercised on its read-failure path. That path returns 0 and prints nothing, which means a broken read is silently reported as "no wedge". A dedicated variable, for example kubectl_wedge_rc, would pin that behaviour.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/run-kubernetes-serial-console_test.bats` around lines 30 - 46, Extend
the *initContainerStatuses* branch of the mock kubectl handler to honor a
dedicated kubectl_wedge_rc return code before printing kubectl_wedge_rows.
Default it to success so existing tests remain unchanged, while allowing tests
to simulate wedge-read failures and verify cozy_report_guest_console_wedge’s
failure behavior.
hack/e2e-chainsaw/_lib/run-kubernetes.sh (1)

754-756: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Quote the expansion in the total-Pod count.

${pods} is unquoted inside printf, so the shell applies pathname expansion in addition to word splitting. A Pod name is unlikely to contain a glob character, but the count is what the truncation marker reports, so make it deterministic. Quoting also clears the SC2086 hint.

♻️ Proposed fix
       printf 'capture stopped after %s Pods; %s matched in total\n' \
-        "${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \
+        "${max_pods}" "$(printf '%s\n' "${pods}" | wc -l | tr -d ' ')" \
         >"${report_dir}/COLLECTION-TRUNCATED.txt"

The count stays the same, because command substitution already stripped the trailing newline from pods.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 754 - 756, Update the
total-Pod count in the truncation report within the collection logic to quote
the pods expansion passed to printf, preventing pathname expansion while
preserving the existing line count and report output.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/run-kubernetes-serial-console_test.bats`:
- Around line 685-688: Update the assertion in the loop around the grep command
to use a negated if condition, so absence of “1m45s” succeeds and its presence
emits the error and returns 1. Preserve the existing failure message and return
behavior.

---

Nitpick comments:
In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh`:
- Around line 754-756: Update the total-Pod count in the truncation report
within the collection logic to quote the pods expansion passed to printf,
preventing pathname expansion while preserving the existing line count and
report output.

In `@hack/run-kubernetes-serial-console_test.bats`:
- Around line 30-46: Extend the *initContainerStatuses* branch of the mock
kubectl handler to honor a dedicated kubectl_wedge_rc return code before
printing kubectl_wedge_rows. Default it to success so existing tests remain
unchanged, while allowing tests to simulate wedge-read failures and verify
cozy_report_guest_console_wedge’s failure behavior.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4fe767b2-283e-438c-8a03-e99c1d11a4bd

📥 Commits

Reviewing files that changed from the base of the PR and between 3c3d6dd and e95dee2.

📒 Files selected for processing (2)
  • hack/e2e-chainsaw/_lib/run-kubernetes.sh
  • hack/run-kubernetes-serial-console_test.bats

Comment on lines +685 to +688
grep -q '1m45s' "$f" && {
echo "$f still carries the pre-Pod-state figure 1m45s" >&2
return 1
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔴 Critical | ⚡ Quick win

This assertion inverts its own result and fails the test on the passing path.

grep -q '1m45s' "$f" && { ...; return 1; } is the last command in each loop iteration. When the file does not contain 1m45s, grep exits 1, the && list is not taken, and the list status is 1. That status becomes the exit status of the for loop, which is the last command in the test, so the test fails exactly when the assertion holds. Use a negated if instead.

🐛 Proposed fix
-    grep -q '1m45s' "$f" && {
-      echo "$f still carries the pre-Pod-state figure 1m45s" >&2
-      return 1
-    }
+    if grep -q '1m45s' "$f"; then
+      echo "$f still carries the pre-Pod-state figure 1m45s" >&2
+      return 1
+    fi
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
grep -q '1m45s' "$f" && {
echo "$f still carries the pre-Pod-state figure 1m45s" >&2
return 1
}
if grep -q '1m45s' "$f"; then
echo "$f still carries the pre-Pod-state figure 1m45s" >&2
return 1
fi
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/run-kubernetes-serial-console_test.bats` around lines 685 - 688, Update
the assertion in the loop around the grep command to use a negated if condition,
so absence of “1m45s” succeeds and its presence emits the error and returns 1.
Preserve the existing failure message and return behavior.

@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the feat/e2e-capture-tenant-serial-console branch from e95dee2 to f724cb9 Compare August 8, 2026 01:40

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
hack/e2e-chainsaw/_lib/run-kubernetes.sh (1)

766-772: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Quote ${pods} in the truncation count.

Shellcheck flags line 769 (SC2086). Pod names cannot contain glob characters, so the current form works, but the quoted form gives the same count and removes the warning.

♻️ Proposed fix
       printf 'capture stopped after %s Pods; %s matched in total\n' \
-        "${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \
+        "${max_pods}" "$(printf '%s\n' "${pods}" | wc -l | tr -d ' ')" \
         >"${report_dir}/COLLECTION-TRUNCATED.txt"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 766 - 772, Update the
truncation count in the pod-collection logic around the break condition to quote
the `${pods}` expansion when passing it to `printf`, preserving the existing
line count while resolving the shellcheck SC2086 warning.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh`:
- Around line 766-772: Update the truncation count in the pod-collection logic
around the break condition to quote the `${pods}` expansion when passing it to
`printf`, preserving the existing line count while resolving the shellcheck
SC2086 warning.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 167038ee-03de-425f-96af-6c183a68788f

📥 Commits

Reviewing files that changed from the base of the PR and between e95dee2 and f724cb9.

📒 Files selected for processing (4)
  • hack/e2e-chainsaw/_lib/run-kubernetes.sh
  • hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml
  • hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml
  • hack/run-kubernetes-serial-console_test.bats
🚧 Files skipped from review as they are similar to previous changes (2)
  • hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml
  • hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml

@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the feat/e2e-capture-tenant-serial-console branch from f724cb9 to 5e784f9 Compare August 8, 2026 02:00

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
hack/e2e-chainsaw/_lib/run-kubernetes.sh (1)

782-784: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Quote ${pods} in the truncation count.

${pods} is already newline-separated, so word splitting is not needed here. Quoting keeps the count correct and clears SC2086.

♻️ Proposed change
       printf 'capture stopped after %s Pods; %s matched in total\n' \
-        "${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \
+        "${max_pods}" "$(printf '%s\n' "${pods}" | wc -l | tr -d ' ')" \
         >"${report_dir}/COLLECTION-TRUNCATED.txt"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh` around lines 782 - 784, Quote the
pods expansion in the truncation-count command within the collection reporting
block, while preserving the existing newline count and whitespace trimming
behavior. Update the printf invocation that feeds wc so it uses the already
newline-separated pods value without word splitting.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@hack/e2e-chainsaw/_lib/run-kubernetes.sh`:
- Around line 782-784: Quote the pods expansion in the truncation-count command
within the collection reporting block, while preserving the existing newline
count and whitespace trimming behavior. Update the printf invocation that feeds
wc so it uses the already newline-separated pods value without word splitting.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3316afdf-222a-4d3c-8c3a-365230681a26

📥 Commits

Reviewing files that changed from the base of the PR and between f724cb9 and 5e784f9.

📒 Files selected for processing (3)
  • hack/e2e-chainsaw/_lib/run-kubernetes.sh
  • hack/run-kubernetes-serial-console_test.bats
  • packages/apps/kubernetes-nodes/tests/render-parity.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • packages/apps/kubernetes-nodes/tests/render-parity.sh

@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the feat/e2e-capture-tenant-serial-console branch from 5e784f9 to 0e1c7c8 Compare August 8, 2026 02:32
The tenant Kubernetes suites fail the node-Ready budget with zero
Nodes registered, and the time is lost inside the guest between the
worker VMI reaching Running and Talos requesting its apid
certificate. Nothing collected today covers that window: the existing
in-guest capture drives talosctl, so it needs the very apid the
worker never reached, and on the management side there is neither a
Node nor a certificate request to read.

Turn on the guest console log for the suite's node group and collect
each virt-launcher's guest-console-log container from the management
cluster. Reads start at the beginning of the stream rather than
tailing it, so a guest that repaints its console cannot push the boot
output out of the window; every read is bounded, and the walk stops
at a cap that records both counts, so the collector cannot become the
unbounded term in a failure path that already runs several others
under one deadline.

A read that returns no console is not filed as a quiet guest. Three
outcomes get three labels, because only one of them says anything
about the guest: the read failed and returned nothing, the read
succeeded and returned nothing, or it broke off part way. Each read's
stderr is kept beside the capture rather than inside it, since
kubectl's own "container is not valid for pod" would otherwise fill
the file and a console that never started would read as one that had
spoken. That case is the headline one here, so the Pod state and
events that explain it are collected alongside, once for the whole
selector.

Enabling the console overrides a cluster-wide setting the platform
holds for an open upstream defect, so the run is also an experiment,
and its own failure mode is the one that hides best: the container
wedges virt-launcher in Init, no worker boots, and the suite reports
"fewer than 2 tenant nodes Ready" -- the same sentence the failure
under study produces. Read as the known signature, the experiment's
answer would be filed as the flake it was meant to explain. So the
failure path names that case first, before any other diagnostic, and
only when it is true.

The passing path checks separately that the container attached at
all, and reports the ratio it saw rather than a bare success, since
the check is deliberately satisfied by any one Pod. It keeps three
outcomes apart too: a read that did not answer, a selector that
matched no Pod, and Pods that matched and carry nothing. It reads the
Pod name alongside the container names so the last two cannot
collapse, and both container lists so a future move of the container
cannot fail a run over a diagnostic. It runs last, after everything
the suite exists to prove, because a debugging aid must not preempt
the assertions that cover the cluster.

The capture runs before the talosctl one because it is the only of
the two that can describe a worker which never reached apid, the
shape this failure keeps taking. kubectl reports a warning and an
empty result alike, on stderr with a zero status, so the exit status
decides which artifact its output lands in.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the feat/e2e-capture-tenant-serial-console branch from 0e1c7c8 to 50c5e2c Compare August 8, 2026 02:48
@lexfrei
Aleksei Sviridkin (lexfrei) merged commit c1071da into main Aug 8, 2026
41 of 43 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the feat/e2e-capture-tenant-serial-console branch August 8, 2026 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/feature Categorizes issue or PR as related to a new feature size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant