test(e2e): measure tenant worker CPU as a rate, and the sandbox nodes under it - #3836

What this PR does

Two instruments for the tenant worker that misses the node-Ready deadline. They are instruments, not a fix: the failure they instrument is #3513, and this PR does not close it. Neither changes what the suites assert.

The worker CPU capture now reads container_cpu_usage_seconds_total beside the CFS counters it already read, and it is read twice a fixed interval apart instead of once. The throttled counters say a container met its ceiling; they cannot say how much CPU it got, and the two failures behind a stalled worker are exactly those, a guest held at a one-core ceiling and a guest whose vCPU thread is rarely scheduled at all. Both leave throttled periods behind. Every counter involved is cumulative since the container started, so a single reading divides out to an average over the whole uptime and says nothing about the window the deadline covered, which matters here because the profile under investigation is a guest that freezes for tens of seconds and then runs flat out: the average comes out low, the instantaneous figure comes out at the cap, and only the second distinguishes it from a guest that is simply capped.

The second instrument is a new subject. The worker captures describe a cgroup, so they are blind to the layer under the sandbox, and a guest that is runnable and never scheduled looks from the cgroup like a guest that asked for nothing. Whether the sandbox node itself was given its turn is recorded only in the steal column of its own /proc/stat. No kubelet endpoint carries it, and crust-gather cannot reach it because PodSecurity refuses its node-shell Pod in this cluster, so the capture goes over the Talos API, which is already how the report reads those nodes for dmesg. It is captured verbatim with its column legend beside it rather than reduced to a percentage: /proc/stat has no header, steal and guest sit next to each other, and guest is large on a node running VMs and already counted inside user, so reading one as the other turns a fraction of a percent into twenty-odd.

Both subjects are read on either side of one shared wait, so the pair pays it once. That wait is guarded like every other external in the block: it is the one call there outside a || true, and under set -eu a missing sleep would exit 127 and take the collectors after it, two of which the phase's own missing-timeout warning promises keep collecting. Without it the two readings are taken back to back and the block says so, which is honest here because the interval is read off the stamps rather than off the knob. The default is 12 seconds and that is the whole wall-clock cost this adds to a failing run. The wait is not the interval either subject's counters span, though: each pass walks its own nodes and the other subject's pass falls between a subject's two readings, and on the run these diagnostics exist for those passes are the slow part rather than the wait. Each capture therefore stamps both ends of its read, so the real interval is subtracted off the pair instead of assumed from the knob, and the width of those brackets is the uncertainty. This matters more on the sandbox side than on the worker side, since a cAdvisor row carries its own sample time and a /proc/stat row carries none. That wait is a measurement interval rather than a stand-in for a condition nobody wrote a wait for, so docs/agents/e2e-testing.md carves it out by name instead of leaving a reader to decide whether the TODO(e2e-replace-fixed-timeouts): marker applies to a gap between two readings of a cumulative counter.

The pair is gated below the serial console rather than above it, and its ceiling rather than its worth is the reason. Reading two collectors twice is sixteen bounded reads plus the interval, which at the read bound and the three-node cap is most of the phase budget, and admission gates only when a collector may start. Above the console, a run that hit those bounds would leave the console never started rather than merely cut short, and the console is the only capture that survives a worker which never reached apid, which is the shape this failure usually takes. That overrun is inside the term the budget derivation already allows for one collector admitted a moment before the deadline, and a guard in hack/run-kubernetes-node-join_test.bats holds the pair's ceiling against it. Two more guards there hold what may sit ahead of the console: nothing read more than once, and nothing whose ceiling could spend the budget on its own.

Failure marking follows the existing collectors rather than inventing a shape: no collector here may leave an empty directory, because an empty directory reads as a healthy node, so a missing binary, a missing talosconfig, a listing that never answered, a node with no address and a read that was cut off each land as a named note or a COLLECTION-FAILED.txt.

The phase budget is unchanged at 420s and the derivation guard still holds.

Covered by hack/run-kubernetes-sandbox-cpu-time_test.bats (new) plus additions to the cpu-throttle, node-join and network-counters suites. Each collector was checked against a deliberate break of its own guard: dropping the usage family, ignoring the sample number, removing the interval, accepting a zero interval, swallowing the missing-talosconfig marker, reading the wrong /proc file, taking a dual-stack node's second address for a node, counting addresses instead of nodes in the truncation marker, stamping only one end of a read, letting the stamp claim counters were sampled on an arm that read nothing, telling a capture with no counters to subtract a sibling, leaving the sampling wait unguarded, and starving the budget guard of its input. Every one is caught by a named test.

Screenshots

Not applicable, no UI change.

Downstream repositories

The diff touches hack/e2e-chainsaw/_lib/run-kubernetes.sh, two chainsaw-test.yaml files, four BATS files and one agent doc. Walking the trigger map: nothing under hack/ is moved or renamed, hack/package.mk and hack/common-envs.mk are untouched, no make target changes what it does, no package is added or removed, no schema, default, enum, CRD or namespace moves. The one entry that mentions this area is the ccp rule about moving or renaming things under hack/, and no file is moved or renamed here.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:

Release note

test(e2e): the node-join failure diagnostics now read the tenant worker CPU counters twice a fixed interval apart, so consumed CPU and throttling are both available as a rate over the same window instead of as an average over the container's uptime, and they add the sandbox node's own /proc/stat over the Talos API, which is the only place the steal time under the sandbox is recorded. Each reading carries the time it was taken, so the interval a rate divides by comes from the captures rather than from the configured wait.

Summary by CodeRabbit

• New Features

  • Enhanced Kubernetes diagnostics with paired, timestamped samples for worker CPU usage and throttling.
  • Added sandbox-node CPU steal-time diagnostics, including support for partial, unavailable, or timed-out captures.
  • Improved diagnostic reports with clearer capture status, rate-calculation context, and sample-specific artifacts.
  • Updated diagnostic sequencing and time budgets to accommodate the expanded metrics collection.

• Documentation

  • Clarified when measurement intervals are appropriate and updated the review checklist accordingly.