test(e2e): price the layers above the sandbox on the node-join window - #3890

What this PR does

Adds two readings to the tenant node-join path, taken on either side of the same wait the KVM counters already bracket: the runner VM kernel's own /proc/stat, and the per-thread CPU time of the QEMU processes the sandbox VMs run as.

Every CPU reading on this failure path has a subject inside the sandbox: a cgroup, a tenant worker's threads, a sandbox node's own /proc/stat. Each of them can say a guest was given its ticks while getting no work done, and none can say what the layers above spent. That is where this suite's own tax is paid. The sandbox nodes' guest time is charged to the runner kernel as user time, and whatever the runner VM loses to the machine hosting it is charged to nobody the sandbox can see, so a node reporting steal 0 and a node whose hypervisor was itself waiting look identical from inside.

/proc/stat is not namespaced, so reading it from the sandbox container reads the runner VM's kernel. That is the only place steal on that layer appears, and it is read in one direction only: a steal that climbs proves this runner VM was preempted, while a zero proves nothing, because the column is filled only where the hypervisor exposes a paravirt steal clock and nothing in the capture observes whether this one does. The sandbox nodes a layer down can make the stronger claim, since this repository starts them with accel=kvm; nobody here starts the runner VM, so for this lane the answer is unknown until somebody reads a non-zero.

The sandbox VMs are QEMU processes in the same PID namespace as the shell driving the suite, so their threads are a local read too. What that adds is not "which node is busy" on its own, since cozy_capture_sandbox_node_cpu_time already reads each node's own /proc/stat by name; it is that split on the join window, since the existing collector samples twice across a twelve-second interval inside the diagnostics block, after the join has already failed. Beyond the window it adds the host-side accounting of each vCPU thread, what the runner actually granted it, which no in-guest counter can report, and QEMU's non-vCPU IO and worker threads, which no in-guest counter sees either. It also names which node each process is, read from the srv<N>/qemu.pid files the bringup already writes: a container pid identifies nothing, and whether the starving guest is the control-plane node or a worker-carrying one is the discriminating question. With no usable pid file the capture stays anonymous and says so, as its KVM sibling does; the pid reads themselves are bounded like every other read here, since a pid file that never yields a byte must not hold the capture.

Both are taken at the two points the KVM counters already use, for the same reason: the values are cumulative, so a rate needs a pair, and the interval worth measuring is the window the guest is failing to boot in rather than a few seconds after the deadline has expired. The thread reading runs the probe the tenant worker capture already uses rather than a copy of it, because that probe already answers the two outcomes that matter here and already names the three ways it can come up short, and a second copy would drift from it exactly at those failure paths.

Each capture carries the layer it describes in its own legend. Three kernels in this report publish a /proc/stat and two captures now read QEMU threads, one layer apart in each direction, so a reader who takes one for the other is comparing a hypervisor against its own guest. The runner-kernel legend also carries the trap specific to that layer: nearly all of its user time IS guest time, because running the sandbox VMs is what that kernel does, so adding the two columns double-counts the whole run.

Cost and blast radius: two bounded local reads per sample, no cluster access, nothing added to any diagnostics-phase budget window. The failure-path calls sit behind the same reasoning as the KVM reading, which the phase-budget guard in hack/run-kubernetes-node-join_test.bats now prices with a cost arm for each; the guards that enumerate collectors (the missing-timeout warning and the documented spend order) carry both new names.

Tests are in hack/run-kubernetes-runner-cpu_test.bats, 44 cases, green under both runner forms. They cover the readings themselves, every legend claim a reader depends on, the bounded read and its knobs, the sample-number split, each way a capture can come up empty (a file with no cpu row, a container with no QEMU, a proc mount that yields nothing, a QEMU named and then gone), the routing of a message from a read that succeeded against one that failed, the per-node attribution and each way it can be incomplete, and placement guards of their own: both pairs must bracket the wait on both paths, nothing may be inserted between the three second readings, and each collector must report a shortfall to the job log in its own words rather than only to the report, since one call site is the passing path where the artifact is the one nobody downloads. Two guards exist because a legend can fail without any number being wrong: one asserts that the parse-from-the-last-parenthesis rule the legend states actually works on the lines the capture holds, and one that every bracketed legend line lands as a single line, since an apostrophe inside one of these single-quoted strings shreds it into one word per line while still passing any short grep.

No documentation change: the KVM pair this mirrors is not described in docs/agents/e2e-testing.md either, and its documentation lives in the legends the captures carry.

Context: #3513

Screenshots

No UI changes.

Downstream repositories

One line of the bringup changes too: hack/e2e-prepare-cluster.bats now starts each sandbox QEMU with -name guest=srv<N>,debug-threads=on. QEMU names vCPU threads (CPU N/KVM) only when asked to, so without the flag every thread reports the bare process name and the vCPU/IO split this capture documents would not exist to read on this lane; a test pins the flag to the legend that relies on it.

Walked the trigger map in docs/agents/contributing.md against the diff. Of the hack/ triggers it names, hack/e2e-prepare-cluster.bats (ansible-cozystack, talm) is touched: the touched lines are the CI sandbox QEMU flags, which neither downstream repository restates -- both provision real machines, not the e2e sandbox. hack/package.mk, hack/update-crd.sh and hack/upload-assets.sh are untouched. The rest of the change is confined to the Chainsaw kubernetes suites' shared library and its unit tests, which no downstream repository restates.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:

Release note

test(e2e): a tenant node-join failure now records the runner kernel's own CPU time and the sandbox VMs' per-thread QEMU CPU time on either side of the join wait, so steal on the layer above the sandbox and an uneven split across the three nodes are both visible in cozyreport.tgz

Summary by CodeRabbit

• New Features

  • Added runner and sandbox CPU diagnostics to node-join troubleshooting captures.
  • Captures now include timing details, layer-specific context, partial-result handling, and explicit failure indicators.
  • Enabled identifiable virtual CPU thread names for improved diagnostic reporting.

• Bug Fixes

  • Improved warning visibility when CPU diagnostic samples are unavailable or time out.
  • Added safer handling for incomplete reads, missing data, and shell errors.

• Tests

  • Expanded coverage for CPU capture behavior, timeout handling, ordering, and failure scenarios.