Skip to content

fix(tests): bound the e2e data-plane capture's reads and stop inferring absence from silence - #3659

Merged
Aleksei Sviridkin (lexfrei) merged 3 commits into
mainfrom
fix/dataplane-capture-bound-reads
Aug 8, 2026
Merged

fix(tests): bound the e2e data-plane capture's reads and stop inferring absence from silence#3659
Aleksei Sviridkin (lexfrei) merged 3 commits into
mainfrom
fix/dataplane-capture-bound-reads

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What this PR does

hack/e2e-capture-dataplane.sh runs after an e2e failure to snapshot the host-to-pod data plane. Every kubectl read in it was unbounded, so a read against a wedged apiserver hung until the caller's own backstop killed the script, and then no capture file was written at all. The run that most needs the diagnostic is the run least likely to produce one. Each read now carries its own bound, 20s for a single read and 28s for a list, well inside the 300s the callers allow.

Bounding the reads creates the second problem, which is most of this diff. A bounded read that gives up returns empty, and empty was already how the script recognised "there is nothing here". Five places turned a read that never answered into a fact about the cluster: no affected pods, no ovn-central, no LoadBalancer Service, no cilium-agent on a node, and a LoadBalancer recorded as reachable. Two of the five only became reachable because of the bounds. Before them the script died instead of writing the file, so a lookup that never answered could not be recorded as a pod that is not there.

Each of those claims now has a counterpart chosen by the status of the read that produced it, and the counterpart says what is left unknown rather than what the read did. The last one is the subtlest, because nothing about it looks like a claim of absence: the probe resolves a cni-server pod before probing, and a lookup that never answered produced no probe outcome at all, which the decision helper read as "nothing failed" and the artifact stamped as reachable. That is a verdict about an address the script never touched. The probe now emits an explicit unknown, and a set of outcomes that is entirely unknown stays unknown. Only a wholly unknown set: one unrun probe beside real failures leaves the failures standing and still captures, because they are the evidence the capture exists to characterise.

Every note ships beside the capture it explains, not only in the job log, because the reader who has the uploaded report and not the run is exactly the reader who cannot otherwise tell a capture that found nothing from one that never ran. Both sibling collectors already hold that contract.

The middle commit is a separate and measurable thing. Each per-node pod lookup was issued up to three times per node; it is asked once now and memoised, with only answers and cutoffs cached, since an instant failure costs nothing to re-ask and caching it would make one transient permanent for the whole run. On an 8-pod, 8-node stub that is 264 kubectl calls down to 229 and 59 per-node lookups down to 24, with byte-identical output.

Deliberately left, and worth naming so nobody reads this as the whole class. The two EndpointSlice reads still discard stderr and report neither outcome. The "reachable, skipped" wording still covers every path that reaches it having run zero probes, among them an unresolved probe node, a _lbport of 0, a cni-server image carrying none of nc, curl or wget, and a failure of the probe exec itself rather than of the lookup before it; that leftover is tracked in #3658. And the three call sites still disagree about what a partial answer means: host_http_probe discards a name that arrived with a non-zero status, while capture_node and capture_lb_node use a non-empty name regardless of it. All of these are verbatim at the merge base and this diff does not change their behaviour.

Covered by 39 cases in hack/capture-dataplane.bats, each branch pinned by a stub that answers one read and fails another, and each with a positive anchor so a stub that stops short of the branch fails rather than passing quietly.

relates to #3642

Screenshots

Downstream repositories

Release note

fix(tests): bound every kubectl read in the e2e data-plane capture so a hung read no longer costs the whole capture, and stop the collector reporting an unanswered read as an absent pod, service or reachable address

Six `kubectl get` calls ran with no bound of their own, while all 27
`kubectl exec` calls in the same script went through one. Against an
apiserver that hangs rather than refuses, the first of them -- the
cluster-wide pod list -- held the script until the caller's backstop
killed it, so the capture wrote nothing at all. From the Chainsaw catch
that is worse than empty: the run is cut off by the outer `timeout -k`
and the script says nothing about having been cut short, so a truncated
capture reads exactly like a complete one that found no affected pods.

Measured against a kubectl stub that never returns: before, the script
was still blocked on its first call when a 60s backstop killed it;
after, it reaches its own end and says which read was cut off and what
that leaves missing. Roughly 4s at a 1s bound, 40s at the 20s default.

A per-read bound buys forward progress, not a smaller total. The sum of
the bounds here is far past any caller's envelope and deliberately so,
because the MAX_PODS and MAX_LBS caps bound work while the outer
`timeout -k` bounds wall clock. What changes is that no single hung read
can consume that envelope before anything is written.

The reporting around them is built to survive being read by someone who
was not there. Only 124 and 137 may name a timeout, since those are the
only statuses a bound produces and every other one is kubectl answering
-- refused, denied, a kind the cluster does not serve -- so the reads
keep their stderr instead of discarding it and quote it back. 137 is
described as what it is, a SIGKILL that cannot be told apart from the
bound's own kill grace. Notes print through `printf` rather than `echo`,
because under /bin/sh echo expands backslash escapes and kubectl's
stderr now flows through the logger: a message holding a literal \n
would otherwise split the note and forge a second "[capture-dataplane]"
line that reads as this script's own verdict, which needs no malice
given the jsonpath in these reads contains {"\n"} and kubectl quotes the
expression back when it fails to parse it.

The per-node lookups ask with `{.items[*].metadata.name}` rather than
`{.items[0]...}`. client-go's evalArray has no allowMissingKeys escape
the way evalField does, so indexing into an empty list is a hard error
and kubectl exits 1 -- which meant nothing while the status was
discarded, and would report "there is no cilium-agent on this node", the
ordinary answer, as a read that failed.

The tests drive the script as a subprocess against a stub kubectl that
hangs, fails instantly, or returns a forged line, and read what it
wrote. The stub that proves forward progress answers the pod list and
hangs afterwards, so the walk over affected pods is actually entered.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The same (namespace, label, node) is asked up to three times per
affected pod across the sections of this script. That was free while the
first call hung forever and the script died inside it; with a per-read
bound each repeat costs its own timeout, and against a wedged apiserver
the repetition becomes the largest consumer of the caller's envelope --
enough that a 300s budget buys two pods where it could buy five.

Answers are memoised, empty ones included: "there is no cilium-agent on
this node" is exactly the lookup worth not repeating.

The memo is a file rather than a variable, which is not a style choice.
The walk over affected pods runs on the right-hand side of a pipeline,
so it executes in a subshell and a variable memo is discarded at the end
of it, leaving every lookup to be paid for again in the sections that
follow. With the variable form the suite still recorded each of the
three lookups twice. That is also why this is not the shape of the node
memo above it, which is a space-delimited string matched with a case
glob and is correct for a dedup living entirely inside the walk.

Keys are compared as whole tab-separated fields, so the '=' inside a
label selector is ordinary payload; what the scheme rests on is that no
namespace, label selector or node name contains a tab, which Kubernetes
names and selectors cannot.

An empty pod list is also separated from one that never answered, so the
branch below it stops reporting a failed read as a cluster with nothing
wrong.

Pinned by a stub that records every per-node lookup it is given: the
test fails if any is issued twice.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Five places turned an empty or missing result into a fact about the
cluster: no affected pods, no ovn-central, no LoadBalancer service, no
cilium-agent or ovs on a node, and a LoadBalancer recorded as reachable.
Each is also what the script produces when the read behind it never
answered, and with a note now saying the read failed, lines sat next to
each other and disagreed.

Each claim has a counterpart chosen by the status of the read that
produced it, and each says what is left unknown rather than what the
read did:

  no scheduled NotReady pods      / whether any pod is affected
  no ovn-central pod in <ns>      / whether <ns> runs ovn-central
  no Service type=LoadBalancer    / whether any LoadBalancer needs capturing
  (no cilium-agent pod found)     / could not determine whether one runs there
  LB reachable, skipped           / whether the LB is reachable is unknown

The last of those is the subtlest, because nothing about it looks like a
claim of absence. host_http_probe resolves a cni-server before probing,
and a lookup that never answered produced no probe outcome at all, which
the decision helper read as "nothing failed" and the artifact stamped as
reachable -- a verdict about an address the script never touched. The
probe now emits an explicit unknown, and a set of outcomes that is
entirely unknown stays unknown rather than collapsing into either
answer. Only a wholly unknown set: one unrun probe beside real failures
leaves the failures standing and still captures, because they are the
evidence the capture exists to characterise.

Wording that describes the read instead would be wrong in a case these
bounds create. A list can fail after emitting rows, since a bound firing
mid-stream leaves output on stdout and 124 in $?, so an empty result
with a non-zero status can mean the read said nothing or that it said
only uninteresting things. Before the bounds a read either answered or
hung forever, and a partial answer was not a state this script could
reach. For the same reason a failure note is unconditional on the status
while the consequence -- skipped, unresolved -- is stated only by the
branch that takes it, which is why the note inside pod_on_node names
none at all: five call sites share it and their consequences differ.

Two of these five are in the artifact rather than the log, and they are
there because of the bounds. Unbounded, the per-node lookup blocked
until the caller's backstop killed the script and no capture file was
written; bounded, the script survives and writes the file, so a read
that never answered would have been recorded as a pod that is not there
or an address that answered. pod_on_node returns its status with the
name and the memo stores it, since a hit that dropped it would launder a
cutoff into a confident absence. The status travels packed with the
value through stdout because every caller reads the helper through a
command substitution, where a variable dies with the subshell -- the
same property that makes the memo itself a file. Only answers and
cutoffs are memoised: an instant failure costs nothing to ask again, and
caching it would make one transient permanent for the whole run.

Every note also ships beside the capture it explains, not only in the
job log, because the reader who has the uploaded report and not the run
is exactly the reader who cannot otherwise tell a capture that found
nothing from one that never ran. Both sibling collectors hold that
contract and this script's notes are modelled on them.

The two phrasing helpers move above the sourcing guard so the unit suite
can assert their branches directly. The stderr sink stays below it,
because anything created above is created again by every test that
sources this file and by every host that exits early for want of
kubectl.

What is deliberately left is the same distinction for the two
EndpointSlice reads, which still discard stderr and report neither
outcome.

Each branch is pinned by a stub that answers one read and fails another,
each partial-answer case by a stub that emits a row and then fails, the
artifact cases in the artifact rather than the log, and the memo's
status round-trip by consuming one lookup from both a miss and a hit.
Every one of them carries a positive anchor as well, so a stub that
stops short of the branch fails rather than passing quietly.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
@github-actions github-actions Bot added size/XL This PR changes 500-999 lines, ignoring generated files area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug labels Aug 8, 2026
@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The dataplane capture script now bounds Kubernetes reads, classifies failures and unknown results, memoizes pod lookups, records diagnostic notes, and cleans temporary files. LoadBalancer probing and capture decisions now handle failed, reachable, unattempted, and unknown outcomes. Bats coverage validates these behaviors.

Changes

Dataplane capture robustness

Layer / File(s) Summary
Bounded reads and diagnostic artifacts
hack/e2e-capture-dataplane.sh, hack/capture-dataplane.bats
The script classifies timeouts and command failures, records cutoff reasons, preserves stderr details, and cleans temporary files. Bats tests cover bounded subprocess behavior and artifact notes.
Resource status and memoization
hack/e2e-capture-dataplane.sh, hack/capture-dataplane.bats
Kubernetes lookups distinguish absent, empty, partial, failed, and unknown results. Repeated pod lookups use file-backed memoization while instantaneous failures remain retryable.
LoadBalancer inventory and probe decisions
hack/e2e-capture-dataplane.sh, hack/capture-dataplane.bats
Service and speaker enumeration use configurable bounds. Probe outcomes support ok, fail, and unknown; failed probes remain capturable, while reachable and wholly unknown cases are skipped or reported unknown.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related issues

Possibly related PRs

  • cozystack/cozystack#3564 — Directly relates to bounded Kubernetes lookups and failed, empty, and partial result handling.
  • cozystack/cozystack#3567 — Directly modifies the same capture script and Bats coverage for timeout and cleanup behavior.
  • cozystack/cozystack#3596 — Relates to bounded reads and diagnostic classification in E2E data collection scripts.

Suggested reviewers: myasnikovdaniil

Sequence Diagram(s)

sequenceDiagram
  participant CaptureScript
  participant KubernetesAPI
  participant CaptureArtifacts
  CaptureScript->>KubernetesAPI: perform bounded resource read
  KubernetesAPI-->>CaptureScript: result, partial output, or failure status
  CaptureScript->>CaptureArtifacts: write capture data and diagnostic notes
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main changes: bounded reads and correct handling of unanswered data-plane capture results.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/dataplane-capture-bound-reads

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
hack/e2e-capture-dataplane.sh (1)

917-918: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

The timeout-absent fallback is inconsistent with the unchanged timeout N kubectl exec calls.

DP_BOUND becomes empty when timeout is not on PATH, so these two EndpointSlice reads then run unbounded. The rest of the script still calls timeout 25 kubectl exec ... and timeout 12 ... literally, so on the same host those calls fail with 127. The fallback therefore protects only the get paths. Consider dropping the fallback and requiring timeout next to the existing command -v kubectl check, or applying the same fallback to the exec calls.

Also applies to: 927-928

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-capture-dataplane.sh` around lines 917 - 918, Make timeout handling
consistent across the script: update the prerequisite checks near the existing
command -v kubectl validation to require timeout, then use the validated timeout
command for the EndpointSlice reads and existing kubectl exec calls. Remove the
DP_BOUND empty fallback so no operations unexpectedly run unbounded.
hack/capture-dataplane.bats (1)

293-299: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

dp_hanging_kubectl_dir has one caller while an identical stub is inlined elsewhere.

The test at lines 754-757 builds the same hanging kubectl stub by hand. Call dp_hanging_kubectl_dir there instead, so the stub exists once.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/capture-dataplane.bats` around lines 293 - 299, Update the test around
the existing inline hanging kubectl setup at lines 754-757 to call
dp_hanging_kubectl_dir instead. Remove the duplicated temporary-directory and
stub creation there while preserving the test’s existing directory variable and
behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/e2e-capture-dataplane.sh`:
- Around line 1054-1062: Ensure the cleanup commands for DP_ERR and _POD_MEMO at
the end of the script cannot determine the script’s exit status. Update both
conditional rm lines to explicitly succeed when their variables are empty or
removal is skipped, preserving the tolerated empty _POD_MEMO behavior and
guaranteeing a zero final status.

---

Nitpick comments:
In `@hack/capture-dataplane.bats`:
- Around line 293-299: Update the test around the existing inline hanging
kubectl setup at lines 754-757 to call dp_hanging_kubectl_dir instead. Remove
the duplicated temporary-directory and stub creation there while preserving the
test’s existing directory variable and behavior.

In `@hack/e2e-capture-dataplane.sh`:
- Around line 917-918: Make timeout handling consistent across the script:
update the prerequisite checks near the existing command -v kubectl validation
to require timeout, then use the validated timeout command for the EndpointSlice
reads and existing kubectl exec calls. Remove the DP_BOUND empty fallback so no
operations unexpectedly run unbounded.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: c1223187-e5e9-4313-8566-b8794ca06196

📥 Commits

Reviewing files that changed from the base of the PR and between c1071da and f923066.

📒 Files selected for processing (2)
  • hack/capture-dataplane.bats
  • hack/e2e-capture-dataplane.sh

Comment on lines +1054 to +1062
# The stderr sink is scratch, not evidence: everything worth keeping from it is
# already quoted into a note above. Removed here rather than from an EXIT trap,
# which this repo's suites ban. This is the only exit reachable once the sink
# exists -- the two earlier ones run before it is created -- so the file leaks
# only when the call-site backstop kills the script mid-run, into the runner's
# TMPDIR.
[ -n "$DP_ERR" ] && rm -f "$DP_ERR"
[ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

The final line can make the script exit non-zero.

Line 1061 is the last command. If mktemp failed earlier, _POD_MEMO is empty, [ -n "$_POD_MEMO" ] returns 1, and the script exits 1. The header at lines 97-101 states that no command can fail the job, and the script handles an empty _POD_MEMO as a tolerated condition everywhere else. Force a zero status on both cleanup lines.

🛠️ Proposed fix
-[ -n "$DP_ERR" ] && rm -f "$DP_ERR"
-[ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO"
+[ -n "$DP_ERR" ] && rm -f "$DP_ERR"
+[ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO"
+exit 0
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# The stderr sink is scratch, not evidence: everything worth keeping from it is
# already quoted into a note above. Removed here rather than from an EXIT trap,
# which this repo's suites ban. This is the only exit reachable once the sink
# exists -- the two earlier ones run before it is created -- so the file leaks
# only when the call-site backstop kills the script mid-run, into the runner's
# TMPDIR.
[ -n "$DP_ERR" ] && rm -f "$DP_ERR"
[ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO"
# The stderr sink is scratch, not evidence: everything worth keeping from it is
# already quoted into a note above. Removed here rather than from an EXIT trap,
# which this repo's suites ban. This is the only exit reachable once the sink
# exists -- the two earlier ones run before it is created -- so the file leaks
# only when the call-site backstop kills the script mid-run, into the runner's
# TMPDIR.
[ -n "$DP_ERR" ] && rm -f "$DP_ERR"
[ -n "$_POD_MEMO" ] && rm -f "$_POD_MEMO"
exit 0
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/e2e-capture-dataplane.sh` around lines 1054 - 1062, Ensure the cleanup
commands for DP_ERR and _POD_MEMO at the end of the script cannot determine the
script’s exit status. Update both conditional rm lines to explicitly succeed
when their variables are empty or removal is skipped, preserving the tolerated
empty _POD_MEMO behavior and guaranteeing a zero final status.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug size/XL This PR changes 500-999 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant