test(e2e): add a runner-layer fixed-work canary around the node-join wait - #3919
Merged
Merged
Conversation
…wait The counters the node-join failure path collects are read in whatever unit their source publishes: time in the CPU rows, events in the KVM exits, instantaneous values and running maxima in the files beside those. Not one of them carries a unit of work of its own, so a machine that spent every tick and got a fraction of the work done reads healthy in all of them. Add a collector that carries its own unit of work. It runs two fixed amounts of work at the runner layer and times each: a loop over a handful of integer-valued scalars, which pressure on the shared cache or the memory controller moves little, and a store stream through blocks larger than the last-level cache one core can allocate into, which is dominated by it. Fixed work gives each reading a scale of its own, so a red run can be read with no green one beside it, and the capture states the expected range and where the estimate comes from. Sampled where the three counter pairs already are, before the wait on every run and after it on both paths, but outside their brackets: it occupies a core rather than reading a file, and a burn inside one of those intervals would land in the interval each pair divides by: runner-kernel CPU time charged to the join window, and for the two counting the guests, a core taken away from what they count. Two arms, each under a twenty-second ceiling with the five-second kill grace behind it, so fifty seconds at worst against the 420s diagnostics budget. The budget guard derives that from the ceiling the source declares, and the phase-warning guard names the canary among the collectors that keep running when timeout is missing. An exit shaped like the ceiling is attributed to it only when the whole bound was on the clock: an outside kill produces the same status, and read by status alone it would publish exactly the finding this collector exists to detect. A reading below the range the legend states, and an arm the ceiling stopped before it could produce one, reach the job log from all three reporting sites, each naming where it was taken, the way a missing reading does from the two outside the diagnostics block. On the passing path the report that holds the figure is the artifact nobody downloads. Each floor is printed in the legend it comes from and sits where a bounded arm crosses it before the ceiling stops the arm, because a floor the ceiling reaches first never fires: the arm is cut off before its rate can fall that far, and every alert then arrives as a bound rather than as the slowdown the floor was meant to name. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
Aleksei Sviridkin (lexfrei)
requested review from
Andrei Kvapil (kvaps),
Timofei Larkin (lllamnyp) and
myasnikovdaniil
as code owners
August 20, 2026 08:25
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughChangesThe Kubernetes e2e diagnostics now include a runner fixed-work canary. It measures fixed compute and memory workloads with optional time bounds, records rates and alerts, and runs samples around the node-join wait. Tests and budget documentation cover the new collector. Runner canary diagnostics
Estimated code review effort: 5 (Critical) | ~90 minutes Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
The diagnostics around the node-join wait already carry counter pairs that report time in the CPU rows, events in the KVM exits, and instantaneous values and running maxima in the files beside those. None of them is a fixed quantity of work, so none can separate a machine that was given fewer turns from a machine that took its turns and got less done with them.
This adds a collector that runs two fixed amounts of work on the runner VM and times them: an integer loop and a streaming write, each under a ceiling of its own. It takes one sample before the node-join wait and one after it, on the failing and the passing path alike. Each sample is an absolute reading rather than half a difference, so one alone is still readable against the expected ranges the capture prints, and the pair says whether a slowdown began inside the interval it brackets or was already there going into it.
The collector sits outside the three counter pairs that bracket the same wait rather than among them, because it occupies a core while it runs, and a burn placed inside one of those intervals would be charged to the join window it is supposed to measure.
A new bats suite of 57 tests covers the collector. The existing node-join guards were extended to price it against the phase budget and to keep the spend order documented in both kubernetes suites in step with the order the block actually runs.
On the budget: the sample taken before the wait is bounded at two arms of the canary ceiling plus grace, so it can add up to fifty seconds ahead of the wait, and the guard prices it at exactly that, while the guard's total stays below an honest worst case because collectors it already priced can spend more than the figures it prices them at.
This branch was reviewed extensively before it was opened, and what is known to be incomplete is written down rather than left implied: the words describing how each floor sits against its healthy band are marked in the code as written rather than derived, so they are the part that can go stale when a constant moves, and the phase budget residuals are tracked in the issue the comments beside them cite.
Screenshots
Not a UI change.
Downstream repositories
Walked the trigger map in
docs/agents/contributing.mdagainst the diff, file by file. The diff is five files: the e2e diagnostics collector library, its two new and extended bats suites, and the two chainsaw suite definitions that document the spend order.The closest call is
cozystack/ccp, whose skills gate onhack/package.mkandhack/common-envs.mkand drivemake generate. Neither file is touched, nothing underhack/is moved or renamed, and no make target changes what it does: the new suite is picked up becauseBATS_UNIT_FILESalready globshack/*.bats, so the file list grows and the target's contract does not.Nothing else matches. No package is added, renamed or removed, so the website app lists are unaffected. No
values.schema.json, version enum, default orApplicationDefinitionchanges, so the Terraform provider is unaffected.hack/e2e-prepare-cluster.batsis untouched, so the node prerequisites thatansible-cozystackandtalmrestate are unchanged.hack/update-crd.sh, therelease.labelsconvention and the chart source kinds are untouched, soexternal-apps-exampleandcozyhrare unaffected. No label, annotation or metric name changes, socozy-proxyandcozystack-telemetry-serverare unaffected. Minimum node and network requirements are unchanged, soexamplesis unaffected.Release note
Summary by CodeRabbit
New Features
Documentation
Tests