fix(e2e): stop the data-plane capture reporting what it never observed - #3672
Conversation
The header credited cozytest.sh alone, while the Chainsaw global catch invokes the same script under a different wall-clock backstop -- 300s against cozytest.sh's 600s. A reader sizing this collector's budget against the caller the header names can still overrun the one it omits, and the arithmetic further down the file already assumes both exist. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
The record filters selected rows by field value and never checked the field count, so a list cut mid-record parsed as a valid record: an IP cut mid-octet still looks like an IP, and a cut inside the nodeName column still looks like a scheduled pod. The capture then probed an address the cluster never had and opened a per-node file for a node that does not exist -- confidently labelled data about objects that are artifacts of the cut. The kubectl jsonpath behind each filter emits a fixed field count per record, so a shorter row is a fragment by construction: seven for the Service list, six for the pod list, five for the endpoint list. Dropping it costs the reader nothing -- the read's own note already says the list did not finish. The endpoint filter is the one that matters most, because it stops at the first row it accepts: a fragment at its head is not one bad row among many, it is the backend, and its half-parsed node is the node every endpoint-side capture then aims at. Where the class lives in this file, so the map does not have to be rebuilt: four sites parse records, and the two loops walking pods and LoadBalancers are not among them -- they consume what these filters already returned, so a cut is caught upstream of them. Three of the four are the filters guarded here. The fourth, the MetalLB speaker loop, is at the limit of what a field count can do: its records carry two fields, a cut in the first leaves no separator and its existing emptiness guard already rejects that, and a cut in the second delivers every separator, which nothing in-band tells apart from a genuinely short node name. That limit is not confined to the speaker loop, and the comment on the first filter now says so rather than implying a guard that holds everywhere: a cut inside any record's LAST field passes a field count. It lands on a display-only column for Services, and on a decision column for the other two, where a truncated value matches neither state being excluded and the row is kept -- over-capturing rather than dropping evidence. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
pod_on_node turns its node argument into `--field-selector spec.nodeName=<node>`, and an empty node asks for pods whose nodeName is empty -- the unscheduled ones. So an unidentified node did not yield "no pod found": it yielded the first pending kube-ovn-cni pod in the namespace, and the tcpdump then ran against a pod that is not on that node and may not be running at all. Both halves of the cross-node capture had this, and each gap opens in the very state its own leg exists to characterise. The announcer is unknown exactly when MetalLB is misbehaving; a Service with no ready endpoint is the ordinary shape of an LB outage, and the endpoint side carried two such lookups, the cni-server pod and the backend's OVS interface. In either state the artifact came out holding a tcpdump from an unrelated pending pod under an ANNOUNCER or ENDPOINT label, or a confusing exec failure -- both pointing away from the actual problem. Each leg is now guarded the way the static capture above it already was, and records in the artifact that it was skipped rather than leaving a silent gap. The guards stay at the call sites rather than moving into pod_on_node: that helper serves five callers, and an early return there would have to invent a status for a lookup nobody made. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two EndpointSlice reads in the LB section discarded stderr and were interpreted by emptiness alone, so a refused or cut-off read wrote the same `<none>` into the artifact that a Service with no endpoints does. That is the assertion without the refutation: a reader holding the uploaded report, and not the run, has no way to tell which happened, and the section is at its weakest exactly when the apiserver is struggling. Both reads now carry their status beside the value, in the vocabulary the per-node captures already use. Reporting the status arms a landmine the silence used to hide: the target port was read with a jsonpath indexing `items[0]`, and client-go's evalArray has no allowMissingKeys escape, so an empty list is a hard error and kubectl exits 1. A Service with no endpointslices is an ordinary answer here and would have started reporting as a failed read, so the read now asks with `items[*]` -- empty output, exit 0 -- and takes the first value, the way the pod lookups already do. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Five inputs walked into the LB probe gate's skip branch having run zero probes, and all five came out as "probe: LB reachable or not probeable ... -- reachable, skipped": no node to probe from, a Service with no port, a probe node the lookup says runs no kube-ovn-cni pod, a host netns with no nc/curl/wget, and an exec that could not run (no timeout on PATH, exec denied, the pod gone mid-probe). None of them observed the address. An e2e image without a probe client would stamp every LoadBalancer in the cluster reachable and skip the whole heavy capture, which reads as a healthy datapath. A sixth way in, the cni-server lookup not answering at all, already reported unknown and still does. Zero probes now routes to that same explicit unknown outcome, and each way of getting there carries a reason the run can actually support. The two structural inputs are named where they are detected. The rest are told apart by the shape of the outcome set, which is the only place the difference survives: the probe helper emits an explicit unknown token only when the lookup itself did not answer, so a set that is present and wholly unknown names the lookup, while an empty set names the three ways a probe dies after the lookup answered. Those three stay together -- the exec's status is swallowed by design, since this collector never fails a job -- but none of them is reported as a lookup that went unanswered, because it did not. The reachable line now says which evidence produced it. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
Both callers of the data-plane collector wrap it in a wall-clock backstop and threw the result away with `|| true`, while the previous-instance logs leg sitting beside each of them records the exit status and prints "INCOMPLETE (exit N); kept what landed in ...". When a backstop killed the collector, nothing said so, and a truncated dataplane/ directory read exactly like a complete one that found little. The collector itself now names every read of its own it could not finish, which left these two call sites as the only remaining way for that leg to die without a word. The Chainsaw one is the tighter of the pair -- 300s against 600s, because it shares an op envelope with the snapshot leg -- so it is both likelier to fire and more expensive to leave silent. In each place the status is read, never propagated: a diagnostic still cannot change a test's outcome. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (5)
📝 WalkthroughWalkthroughThe PR improves dataplane capture accuracy and completion reporting. It rejects truncated records, distinguishes unknown and unreachable states, guards node selection, preserves partial output, and reports timeout or collector failures from cozytest and Chainsaw. ChangesDataplane capture behavior
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
What this PR does
Six fixes to the e2e host->pod data-plane collector and the two call sites that invoke it, all of one class: a diagnostic that states a fact it never observed. Every one of them makes a failed run look healthier than it was, which is the worst direction for a tool that only ever runs on an already-failed test.
awk -F'|'row filters selected by field value and never checked the field count, so an IP cut mid-octet still looked like an IP and a cut inside a nodeName still looked like a scheduled pod. The capture then probed an address the cluster never had and opened a per-node file for a node that does not exist. Each jsonpath emits a fixed field count per record, so a shorter row is a fragment by construction.kube-ovn-cnipod, a host netns with nonc/curl/wget, and an exec that could not run. An image without a probe client alone would stamp every LoadBalancer in the cluster reachable and skip the whole heavy capture. They now report unprobed, each with a reason the run can actually support.<none>a Service with no endpoints writes. They now carry their status. That change armed a landmine the silence had hidden: the target port was read with{.items[0]...}, and client-go'sevalArrayhas noallowMissingKeysescape, so an endpoint-less Service is a hard error rather than an empty answer. The read now asks with{.items[*]...}.pod_on_nodeturns its node argument into--field-selector spec.nodeName=<node>, so an empty node asks for pods whose nodeName is empty, i.e. unscheduled ones. Both tcpdump legs could hand a pending pod to a capture stamped ANNOUNCER or ENDPOINT. Each leg is now guarded and says in the artifact that it was skipped.dataplane/directory was indistinguishable from a complete one that found little.Diagnostics only. No retries, no cluster mutation, no change to any test's pass/fail outcome: the statuses that are now read are used to print a line and nothing else.
Every behavioural change is pinned by a test in the existing bats surface, verified red against the committed tree before the fix. The two suites run 53 and 5 tests.
Known limits and deliberate omissions, named so the next reader does not have to rediscover them:
speaker-1|nodis indistinguishable from a genuinely short node name, so a count there would reject nothing while reading as a closed gap.INCOMPLETE (exit N). Exit 127 (notimeouton PATH) means nothing ran, so "kept what landed" overstates it. Left alone deliberately: the same wording ships on the previous-logs and crust-gather legs beside them, and splitting only the two new ones would create the asymmetry this PR exists to remove. All four want the same treatment together.hack/capture-dataplane.batsnow needsyqfor one test, and unlike three sibling suites it does not declare that at the top of the file.The fan-out caps are sized in units of work while the backstops are wall-clock, and the worst case exceeds both by multiples. That predates this branch and is unchanged by it, but the new INCOMPLETE line is what will make it visible in an artifact for the first time. Tracked separately in #3671.
relates to #3658 #3660 #3654 #3657 #3663 #3649
Screenshots
Downstream repositories
Release note
Summary by CodeRabbit
Bug Fixes
Tests