fix(tests): bound the e2e data-plane capture's reads and stop inferring absence from silence - #3659
Conversation
Six `kubectl get` calls ran with no bound of their own, while all 27
`kubectl exec` calls in the same script went through one. Against an
apiserver that hangs rather than refuses, the first of them -- the
cluster-wide pod list -- held the script until the caller's backstop
killed it, so the capture wrote nothing at all. From the Chainsaw catch
that is worse than empty: the run is cut off by the outer `timeout -k`
and the script says nothing about having been cut short, so a truncated
capture reads exactly like a complete one that found no affected pods.
Measured against a kubectl stub that never returns: before, the script
was still blocked on its first call when a 60s backstop killed it;
after, it reaches its own end and says which read was cut off and what
that leaves missing. Roughly 4s at a 1s bound, 40s at the 20s default.
A per-read bound buys forward progress, not a smaller total. The sum of
the bounds here is far past any caller's envelope and deliberately so,
because the MAX_PODS and MAX_LBS caps bound work while the outer
`timeout -k` bounds wall clock. What changes is that no single hung read
can consume that envelope before anything is written.
The reporting around them is built to survive being read by someone who
was not there. Only 124 and 137 may name a timeout, since those are the
only statuses a bound produces and every other one is kubectl answering
-- refused, denied, a kind the cluster does not serve -- so the reads
keep their stderr instead of discarding it and quote it back. 137 is
described as what it is, a SIGKILL that cannot be told apart from the
bound's own kill grace. Notes print through `printf` rather than `echo`,
because under /bin/sh echo expands backslash escapes and kubectl's
stderr now flows through the logger: a message holding a literal \n
would otherwise split the note and forge a second "[capture-dataplane]"
line that reads as this script's own verdict, which needs no malice
given the jsonpath in these reads contains {"\n"} and kubectl quotes the
expression back when it fails to parse it.
The per-node lookups ask with `{.items[*].metadata.name}` rather than
`{.items[0]...}`. client-go's evalArray has no allowMissingKeys escape
the way evalField does, so indexing into an empty list is a hard error
and kubectl exits 1 -- which meant nothing while the status was
discarded, and would report "there is no cilium-agent on this node", the
ordinary answer, as a read that failed.
The tests drive the script as a subprocess against a stub kubectl that
hangs, fails instantly, or returns a forged line, and read what it
wrote. The stub that proves forward progress answers the pod list and
hangs afterwards, so the walk over affected pods is actually entered.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The same (namespace, label, node) is asked up to three times per affected pod across the sections of this script. That was free while the first call hung forever and the script died inside it; with a per-read bound each repeat costs its own timeout, and against a wedged apiserver the repetition becomes the largest consumer of the caller's envelope -- enough that a 300s budget buys two pods where it could buy five. Answers are memoised, empty ones included: "there is no cilium-agent on this node" is exactly the lookup worth not repeating. The memo is a file rather than a variable, which is not a style choice. The walk over affected pods runs on the right-hand side of a pipeline, so it executes in a subshell and a variable memo is discarded at the end of it, leaving every lookup to be paid for again in the sections that follow. With the variable form the suite still recorded each of the three lookups twice. That is also why this is not the shape of the node memo above it, which is a space-delimited string matched with a case glob and is correct for a dedup living entirely inside the walk. Keys are compared as whole tab-separated fields, so the '=' inside a label selector is ordinary payload; what the scheme rests on is that no namespace, label selector or node name contains a tab, which Kubernetes names and selectors cannot. An empty pod list is also separated from one that never answered, so the branch below it stops reporting a failed read as a cluster with nothing wrong. Pinned by a stub that records every per-node lookup it is given: the test fails if any is issued twice. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Five places turned an empty or missing result into a fact about the cluster: no affected pods, no ovn-central, no LoadBalancer service, no cilium-agent or ovs on a node, and a LoadBalancer recorded as reachable. Each is also what the script produces when the read behind it never answered, and with a note now saying the read failed, lines sat next to each other and disagreed. Each claim has a counterpart chosen by the status of the read that produced it, and each says what is left unknown rather than what the read did: no scheduled NotReady pods / whether any pod is affected no ovn-central pod in <ns> / whether <ns> runs ovn-central no Service type=LoadBalancer / whether any LoadBalancer needs capturing (no cilium-agent pod found) / could not determine whether one runs there LB reachable, skipped / whether the LB is reachable is unknown The last of those is the subtlest, because nothing about it looks like a claim of absence. host_http_probe resolves a cni-server before probing, and a lookup that never answered produced no probe outcome at all, which the decision helper read as "nothing failed" and the artifact stamped as reachable -- a verdict about an address the script never touched. The probe now emits an explicit unknown, and a set of outcomes that is entirely unknown stays unknown rather than collapsing into either answer. Only a wholly unknown set: one unrun probe beside real failures leaves the failures standing and still captures, because they are the evidence the capture exists to characterise. Wording that describes the read instead would be wrong in a case these bounds create. A list can fail after emitting rows, since a bound firing mid-stream leaves output on stdout and 124 in $?, so an empty result with a non-zero status can mean the read said nothing or that it said only uninteresting things. Before the bounds a read either answered or hung forever, and a partial answer was not a state this script could reach. For the same reason a failure note is unconditional on the status while the consequence -- skipped, unresolved -- is stated only by the branch that takes it, which is why the note inside pod_on_node names none at all: five call sites share it and their consequences differ. Two of these five are in the artifact rather than the log, and they are there because of the bounds. Unbounded, the per-node lookup blocked until the caller's backstop killed the script and no capture file was written; bounded, the script survives and writes the file, so a read that never answered would have been recorded as a pod that is not there or an address that answered. pod_on_node returns its status with the name and the memo stores it, since a hit that dropped it would launder a cutoff into a confident absence. The status travels packed with the value through stdout because every caller reads the helper through a command substitution, where a variable dies with the subshell -- the same property that makes the memo itself a file. Only answers and cutoffs are memoised: an instant failure costs nothing to ask again, and caching it would make one transient permanent for the whole run. Every note also ships beside the capture it explains, not only in the job log, because the reader who has the uploaded report and not the run is exactly the reader who cannot otherwise tell a capture that found nothing from one that never ran. Both sibling collectors hold that contract and this script's notes are modelled on them. The two phrasing helpers move above the sourcing guard so the unit suite can assert their branches directly. The stderr sink stays below it, because anything created above is created again by every test that sources this file and by every host that exits early for want of kubectl. What is deliberately left is the same distinction for the two EndpointSlice reads, which still discard stderr and report neither outcome. Each branch is pinned by a stub that answers one read and fails another, each partial-answer case by a stub that emits a row and then fails, the artifact cases in the artifact rather than the log, and the memo's status round-trip by consuming one lookup from both a miss and a hit. Every one of them carries a positive anchor as well, so a stub that stops short of the branch fails rather than passing quietly. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
📝 WalkthroughWalkthroughThe dataplane capture script now bounds Kubernetes reads, classifies failures and unknown results, memoizes pod lookups, records diagnostic notes, and cleans temporary files. LoadBalancer probing and capture decisions now handle failed, reachable, unattempted, and unknown outcomes. Bats coverage validates these behaviors. ChangesDataplane capture robustness
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant CaptureScript
participant KubernetesAPI
participant CaptureArtifacts
CaptureScript->>KubernetesAPI: perform bounded resource read
KubernetesAPI-->>CaptureScript: result, partial output, or failure status
CaptureScript->>CaptureArtifacts: write capture data and diagnostic notes
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
hack/e2e-capture-dataplane.sh (1)
917-918: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueThe
timeout-absent fallback is inconsistent with the unchangedtimeout N kubectl execcalls.
DP_BOUNDbecomes empty whentimeoutis not on PATH, so these two EndpointSlice reads then run unbounded. The rest of the script still callstimeout 25 kubectl exec ...andtimeout 12 ...literally, so on the same host those calls fail with 127. The fallback therefore protects only thegetpaths. Consider dropping the fallback and requiringtimeoutnext to the existingcommand -v kubectlcheck, or applying the same fallback to theexeccalls.Also applies to: 927-928
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/e2e-capture-dataplane.sh` around lines 917 - 918, Make timeout handling consistent across the script: update the prerequisite checks near the existing command -v kubectl validation to require timeout, then use the validated timeout command for the EndpointSlice reads and existing kubectl exec calls. Remove the DP_BOUND empty fallback so no operations unexpectedly run unbounded.hack/capture-dataplane.bats (1)
293-299: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
dp_hanging_kubectl_dirhas one caller while an identical stub is inlined elsewhere.The test at lines 754-757 builds the same hanging
kubectlstub by hand. Calldp_hanging_kubectl_dirthere instead, so the stub exists once.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@hack/capture-dataplane.bats` around lines 293 - 299, Update the test around the existing inline hanging kubectl setup at lines 754-757 to call dp_hanging_kubectl_dir instead. Remove the duplicated temporary-directory and stub creation there while preserving the test’s existing directory variable and behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/e2e-capture-dataplane.sh`:
- Around line 1054-1062: Ensure the cleanup commands for DP_ERR and _POD_MEMO at
the end of the script cannot determine the script’s exit status. Update both
conditional rm lines to explicitly succeed when their variables are empty or
removal is skipped, preserving the tolerated empty _POD_MEMO behavior and
guaranteeing a zero final status.
---
Nitpick comments:
In `@hack/capture-dataplane.bats`:
- Around line 293-299: Update the test around the existing inline hanging
kubectl setup at lines 754-757 to call dp_hanging_kubectl_dir instead. Remove
the duplicated temporary-directory and stub creation there while preserving the
test’s existing directory variable and behavior.
In `@hack/e2e-capture-dataplane.sh`:
- Around line 917-918: Make timeout handling consistent across the script:
update the prerequisite checks near the existing command -v kubectl validation
to require timeout, then use the validated timeout command for the EndpointSlice
reads and existing kubectl exec calls. Remove the DP_BOUND empty fallback so no
operations unexpectedly run unbounded.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: c1223187-e5e9-4313-8566-b8794ca06196
📒 Files selected for processing (2)
hack/capture-dataplane.batshack/e2e-capture-dataplane.sh
| # The stderr sink is scratch, not evidence: everything worth keeping from it is | ||
| # already quoted into a note above. Removed here rather than from an EXIT trap, | ||
| # which this repo's suites ban. This is the only exit reachable once the sink | ||
| # exists -- the two earlier ones run before it is created -- so the file leaks | ||
| # only when the call-site backstop kills the script mid-run, into the runner's | ||
| # TMPDIR. | ||
| [ -n "$DP_ERR" ] && rm -f "$DP_ERR" | ||
| [ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO" | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
The final line can make the script exit non-zero.
Line 1061 is the last command. If mktemp failed earlier, _POD_MEMO is empty, [ -n "$_POD_MEMO" ] returns 1, and the script exits 1. The header at lines 97-101 states that no command can fail the job, and the script handles an empty _POD_MEMO as a tolerated condition everywhere else. Force a zero status on both cleanup lines.
🛠️ Proposed fix
-[ -n "$DP_ERR" ] && rm -f "$DP_ERR"
-[ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO"
+[ -n "$DP_ERR" ] && rm -f "$DP_ERR"
+[ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO"
+exit 0📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # The stderr sink is scratch, not evidence: everything worth keeping from it is | |
| # already quoted into a note above. Removed here rather than from an EXIT trap, | |
| # which this repo's suites ban. This is the only exit reachable once the sink | |
| # exists -- the two earlier ones run before it is created -- so the file leaks | |
| # only when the call-site backstop kills the script mid-run, into the runner's | |
| # TMPDIR. | |
| [ -n "$DP_ERR" ] && rm -f "$DP_ERR" | |
| [ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO" | |
| # The stderr sink is scratch, not evidence: everything worth keeping from it is | |
| # already quoted into a note above. Removed here rather than from an EXIT trap, | |
| # which this repo's suites ban. This is the only exit reachable once the sink | |
| # exists -- the two earlier ones run before it is created -- so the file leaks | |
| # only when the call-site backstop kills the script mid-run, into the runner's | |
| # TMPDIR. | |
| [ -n "$DP_ERR" ] && rm -f "$DP_ERR" | |
| [ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO" | |
| exit 0 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/e2e-capture-dataplane.sh` around lines 1054 - 1062, Ensure the cleanup
commands for DP_ERR and _POD_MEMO at the end of the script cannot determine the
script’s exit status. Update both conditional rm lines to explicitly succeed
when their variables are empty or removal is skipped, preserving the tolerated
empty _POD_MEMO behavior and guaranteeing a zero final status.
What this PR does
hack/e2e-capture-dataplane.shruns after an e2e failure to snapshot the host-to-pod data plane. Everykubectlread in it was unbounded, so a read against a wedged apiserver hung until the caller's own backstop killed the script, and then no capture file was written at all. The run that most needs the diagnostic is the run least likely to produce one. Each read now carries its own bound, 20s for a single read and 28s for a list, well inside the 300s the callers allow.Bounding the reads creates the second problem, which is most of this diff. A bounded read that gives up returns empty, and empty was already how the script recognised "there is nothing here". Five places turned a read that never answered into a fact about the cluster: no affected pods, no ovn-central, no LoadBalancer Service, no cilium-agent on a node, and a LoadBalancer recorded as reachable. Two of the five only became reachable because of the bounds. Before them the script died instead of writing the file, so a lookup that never answered could not be recorded as a pod that is not there.
Each of those claims now has a counterpart chosen by the status of the read that produced it, and the counterpart says what is left unknown rather than what the read did. The last one is the subtlest, because nothing about it looks like a claim of absence: the probe resolves a cni-server pod before probing, and a lookup that never answered produced no probe outcome at all, which the decision helper read as "nothing failed" and the artifact stamped as reachable. That is a verdict about an address the script never touched. The probe now emits an explicit unknown, and a set of outcomes that is entirely unknown stays unknown. Only a wholly unknown set: one unrun probe beside real failures leaves the failures standing and still captures, because they are the evidence the capture exists to characterise.
Every note ships beside the capture it explains, not only in the job log, because the reader who has the uploaded report and not the run is exactly the reader who cannot otherwise tell a capture that found nothing from one that never ran. Both sibling collectors already hold that contract.
The middle commit is a separate and measurable thing. Each per-node pod lookup was issued up to three times per node; it is asked once now and memoised, with only answers and cutoffs cached, since an instant failure costs nothing to re-ask and caching it would make one transient permanent for the whole run. On an 8-pod, 8-node stub that is 264 kubectl calls down to 229 and 59 per-node lookups down to 24, with byte-identical output.
Deliberately left, and worth naming so nobody reads this as the whole class. The two EndpointSlice reads still discard stderr and report neither outcome. The "reachable, skipped" wording still covers every path that reaches it having run zero probes, among them an unresolved probe node, a
_lbportof 0, a cni-server image carrying none ofnc,curlorwget, and a failure of the probeexecitself rather than of the lookup before it; that leftover is tracked in #3658. And the three call sites still disagree about what a partial answer means:host_http_probediscards a name that arrived with a non-zero status, whilecapture_nodeandcapture_lb_nodeuse a non-empty name regardless of it. All of these are verbatim at the merge base and this diff does not change their behaviour.Covered by 39 cases in
hack/capture-dataplane.bats, each branch pinned by a stub that answers one read and fails another, and each with a positive anchor so a stub that stops short of the branch fails rather than passing quietly.relates to #3642
Screenshots
Downstream repositories
Release note