Skip to content

fix(e2e): stop the data-plane capture reporting what it never observed - #3672

Merged
Aleksei Sviridkin (lexfrei) merged 6 commits into
mainfrom
fix/dataplane-capture-hardening
Aug 8, 2026
Merged

fix(e2e): stop the data-plane capture reporting what it never observed#3672
Aleksei Sviridkin (lexfrei) merged 6 commits into
mainfrom
fix/dataplane-capture-hardening

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Six fixes to the e2e host->pod data-plane collector and the two call sites that invoke it, all of one class: a diagnostic that states a fact it never observed. Every one of them makes a failed run look healthier than it was, which is the worst direction for a tool that only ever runs on an already-failed test.

  • A list cut mid-record parsed as a valid record. Three awk -F'|' row filters selected by field value and never checked the field count, so an IP cut mid-octet still looked like an IP and a cut inside a nodeName still looked like a scheduled pod. The capture then probed an address the cluster never had and opened a per-node file for a node that does not exist. Each jsonpath emits a fixed field count per record, so a shorter row is a fragment by construction.
  • Five distinct paths ran zero probes and all reported "reachable, skipped": no node to probe from, a Service with no port, a probe node with no kube-ovn-cni pod, a host netns with no nc/curl/wget, and an exec that could not run. An image without a probe client alone would stamp every LoadBalancer in the cluster reachable and skip the whole heavy capture. They now report unprobed, each with a reason the run can actually support.
  • Both EndpointSlice reads discarded stderr and were read by emptiness alone, so a refused read wrote the same <none> a Service with no endpoints writes. They now carry their status. That change armed a landmine the silence had hidden: the target port was read with {.items[0]...}, and client-go's evalArray has no allowMissingKeys escape, so an endpoint-less Service is a hard error rather than an empty answer. The read now asks with {.items[*]...}.
  • pod_on_node turns its node argument into --field-selector spec.nodeName=<node>, so an empty node asks for pods whose nodeName is empty, i.e. unscheduled ones. Both tcpdump legs could hand a pending pod to a capture stamped ANNOUNCER or ENDPOINT. Each leg is now guarded and says in the artifact that it was skipped.
  • Both callers wrapped the collector in a wall-clock backstop and discarded the result, while the previous-logs leg beside each of them reports "INCOMPLETE (exit N)". A truncated dataplane/ directory was indistinguishable from a complete one that found little.
  • The file's own header credited one caller where two exist, under different backstops (600s and 300s), so a change sized against the named one can overrun the omitted one.

Diagnostics only. No retries, no cluster mutation, no change to any test's pass/fail outcome: the statuses that are now read are used to print a line and nothing else.

Every behavioural change is pinned by a test in the existing bats surface, verified red against the committed tree before the fix. The two suites run 53 and 5 tests.

Known limits and deliberate omissions, named so the next reader does not have to rediscover them:

  • A field count cannot catch a cut inside a record's LAST field, since every separator has already arrived. The comment on the first filter says so and says where it lands for each of the three. The MetalLB speaker loop is left without a count for the same reason: its records carry two fields, and speaker-1|nod is indistinguishable from a genuinely short node name, so a count there would reject nothing while reading as a closed gap.
  • The backstop lines report any non-zero status as INCOMPLETE (exit N). Exit 127 (no timeout on PATH) means nothing ran, so "kept what landed" overstates it. Left alone deliberately: the same wording ships on the previous-logs and crust-gather legs beside them, and splitting only the two new ones would create the asymmetry this PR exists to remove. All four want the same treatment together.
  • When a probe's cni-server lookup fails once and then succeeds while no probe client is present, the reason line names the lookup for all three attempts. The named cause did occur, so nothing false reaches the artifact, but it is narrower than what happened.
  • hack/capture-dataplane.bats now needs yq for one test, and unlike three sibling suites it does not declare that at the top of the file.

The fan-out caps are sized in units of work while the backstops are wall-clock, and the worst case exceeds both by multiples. That predates this branch and is unchanged by it, but the new INCOMPLETE line is what will make it visible in an artifact for the first time. Tracked separately in #3671.

relates to #3658 #3660 #3654 #3657 #3663 #3649

Screenshots

Downstream repositories

Release note

fix(e2e): the host->pod data-plane capture no longer reports absence or reachability it never observed -- truncated list records are dropped instead of parsed, a LoadBalancer no probe reached is recorded as unprobed rather than reachable, an unread EndpointSlice backend is distinguished from an absent one, a tcpdump leg whose node was never identified is skipped instead of aimed at an unscheduled pod, and both callers report a capture their backstop cut short

Summary by CodeRabbit

  • Bug Fixes

    • Improved data-plane capture reliability and reporting for timeouts, failed or partial reads, and missing collectors.
    • Prevented incomplete records and unknown probe results from being treated as valid or reachable.
    • Avoided capturing from unintended nodes when node information is unavailable.
    • Preserved partial diagnostic output while clearly reporting incomplete captures.
  • Tests

    • Expanded coverage for capture outcomes, artifact accuracy, timeout handling, and guarded node selection.

The header credited cozytest.sh alone, while the Chainsaw global catch
invokes the same script under a different wall-clock backstop -- 300s
against cozytest.sh's 600s. A reader sizing this collector's budget
against the caller the header names can still overrun the one it omits,
and the arithmetic further down the file already assumes both exist.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The record filters selected rows by field value and never checked the
field count, so a list cut mid-record parsed as a valid record: an IP cut
mid-octet still looks like an IP, and a cut inside the nodeName column
still looks like a scheduled pod. The capture then probed an address the
cluster never had and opened a per-node file for a node that does not
exist -- confidently labelled data about objects that are artifacts of
the cut.

The kubectl jsonpath behind each filter emits a fixed field count per
record, so a shorter row is a fragment by construction: seven for the
Service list, six for the pod list, five for the endpoint list. Dropping
it costs the reader nothing -- the read's own note already says the list
did not finish. The endpoint filter is the one that matters most, because
it stops at the first row it accepts: a fragment at its head is not one
bad row among many, it is the backend, and its half-parsed node is the
node every endpoint-side capture then aims at.

Where the class lives in this file, so the map does not have to be
rebuilt: four sites parse records, and the two loops walking pods and
LoadBalancers are not among them -- they consume what these filters
already returned, so a cut is caught upstream of them. Three of the four
are the filters guarded here. The fourth, the MetalLB speaker loop, is at
the limit of what a field count can do: its records carry two fields, a
cut in the first leaves no separator and its existing emptiness guard
already rejects that, and a cut in the second delivers every separator,
which nothing in-band tells apart from a genuinely short node name.

That limit is not confined to the speaker loop, and the comment on the
first filter now says so rather than implying a guard that holds
everywhere: a cut inside any record's LAST field passes a field count. It
lands on a display-only column for Services, and on a decision column for
the other two, where a truncated value matches neither state being
excluded and the row is kept -- over-capturing rather than dropping
evidence.

Assisted-By: Claude <noreply@anthropic.com>

Signed-off-by: Aleksei Sviridkin <f@lex.la>
pod_on_node turns its node argument into `--field-selector
spec.nodeName=<node>`, and an empty node asks for pods whose nodeName is
empty -- the unscheduled ones. So an unidentified node did not yield "no
pod found": it yielded the first pending kube-ovn-cni pod in the
namespace, and the tcpdump then ran against a pod that is not on that
node and may not be running at all.

Both halves of the cross-node capture had this, and each gap opens in
the very state its own leg exists to characterise. The announcer is
unknown exactly when MetalLB is misbehaving; a Service with no ready
endpoint is the ordinary shape of an LB outage, and the endpoint side
carried two such lookups, the cni-server pod and the backend's OVS
interface. In either state the artifact came out holding a tcpdump from
an unrelated pending pod under an ANNOUNCER or ENDPOINT label, or a
confusing exec failure -- both pointing away from the actual problem.

Each leg is now guarded the way the static capture above it already was,
and records in the artifact that it was skipped rather than leaving a
silent gap. The guards stay at the call sites rather than moving into
pod_on_node: that helper serves five callers, and an early return there
would have to invent a status for a lookup nobody made.

Assisted-By: Claude <noreply@anthropic.com>

Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two EndpointSlice reads in the LB section discarded stderr and were
interpreted by emptiness alone, so a refused or cut-off read wrote the
same `<none>` into the artifact that a Service with no endpoints does.
That is the assertion without the refutation: a reader holding the
uploaded report, and not the run, has no way to tell which happened, and
the section is at its weakest exactly when the apiserver is struggling.
Both reads now carry their status beside the value, in the vocabulary the
per-node captures already use.

Reporting the status arms a landmine the silence used to hide: the target
port was read with a jsonpath indexing `items[0]`, and client-go's
evalArray has no allowMissingKeys escape, so an empty list is a hard
error and kubectl exits 1. A Service with no endpointslices is an
ordinary answer here and would have started reporting as a failed read,
so the read now asks with `items[*]` -- empty output, exit 0 -- and takes
the first value, the way the pod lookups already do.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Five inputs walked into the LB probe gate's skip branch having run zero
probes, and all five came out as "probe: LB reachable or not probeable
... -- reachable, skipped": no node to probe from, a Service with no
port, a probe node the lookup says runs no kube-ovn-cni pod, a host netns
with no nc/curl/wget, and an exec that could not run (no timeout on PATH,
exec denied, the pod gone mid-probe). None of them observed the address.
An e2e image without a probe client would stamp every LoadBalancer in the
cluster reachable and skip the whole heavy capture, which reads as a
healthy datapath. A sixth way in, the cni-server lookup not answering at
all, already reported unknown and still does.

Zero probes now routes to that same explicit unknown outcome, and each
way of getting there carries a reason the run can actually support. The
two structural inputs are named where they are detected. The rest are
told apart by the shape of the outcome set, which is the only place the
difference survives: the probe helper emits an explicit unknown token
only when the lookup itself did not answer, so a set that is present and
wholly unknown names the lookup, while an empty set names the three ways
a probe dies after the lookup answered. Those three stay together --
the exec's status is swallowed by design, since this collector never
fails a job -- but none of them is reported as a lookup that went
unanswered, because it did not. The reachable line now says which
evidence produced it.

Assisted-By: Claude <noreply@anthropic.com>

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Both callers of the data-plane collector wrap it in a wall-clock backstop
and threw the result away with `|| true`, while the previous-instance
logs leg sitting beside each of them records the exit status and prints
"INCOMPLETE (exit N); kept what landed in ...". When a backstop killed
the collector, nothing said so, and a truncated dataplane/ directory read
exactly like a complete one that found little.

The collector itself now names every read of its own it could not finish,
which left these two call sites as the only remaining way for that leg to
die without a word. The Chainsaw one is the tighter of the pair -- 300s
against 600s, because it shares an op envelope with the snapshot leg --
so it is both likelier to fire and more expensive to leave silent. In
each place the status is read, never propagated: a diagnostic still
cannot change a test's outcome.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
@github-actions github-actions Bot added size/XL This PR changes 500-999 lines, ignoring generated files area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug labels Aug 8, 2026
@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7554cdd4-a7a0-4286-bfc9-46f3e84362be

📥 Commits

Reviewing files that changed from the base of the PR and between 92ec49e and c696154.

📒 Files selected for processing (5)
  • hack/capture-dataplane.bats
  • hack/cozytest-capture-gate.bats
  • hack/cozytest.sh
  • hack/e2e-capture-dataplane.sh
  • hack/e2e-chainsaw/.chainsaw.yaml

📝 Walkthrough

Walkthrough

The PR improves dataplane capture accuracy and completion reporting. It rejects truncated records, distinguishes unknown and unreachable states, guards node selection, preserves partial output, and reports timeout or collector failures from cozytest and Chainsaw.

Changes

Dataplane capture behavior

Layer / File(s) Summary
Read validation and status tracking
hack/e2e-capture-dataplane.sh, hack/capture-dataplane.bats
Parsing rejects truncated service, pod, and EndpointSlice records. Reads distinguish partial, failed, empty, and unknown results.
Probe decisions and guarded capture selection
hack/e2e-capture-dataplane.sh, hack/capture-dataplane.bats
LoadBalancer decisions distinguish reachable, unprobed, and capture-required states. Unknown nodes no longer produce invalid selectors or tcpdump artifacts.
Caller completion reporting and validation
hack/cozytest.sh, hack/e2e-chainsaw/.chainsaw.yaml, hack/cozytest-capture-gate.bats, hack/capture-dataplane.bats
Capture callers record exit statuses, report incomplete captures, preserve partial output, and test timeout and missing-collector behavior.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related issues

Possibly related PRs

Suggested reviewers: myasnikovdaniil

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: preventing data-plane capture from reporting observations it did not make.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/dataplane-capture-hardening

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@lexfrei
Aleksei Sviridkin (lexfrei) merged commit 1b52832 into main Aug 8, 2026
15 of 17 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the fix/dataplane-capture-hardening branch August 8, 2026 17:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/testing Issues or PRs related to testing (e2e, bats, unit tests) kind/bug Categorizes issue or PR as related to a bug size/XL This PR changes 500-999 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant