fix(ci): close the silent-green gaps in e2e selection and the helm unit sweep - #3670
Conversation
Fourteen tests read `[ "$(echo "$output" | wc -w)" -gt 5 ]` to mean "the full suite was selected". A word count clears that bar on any selection of six or more, and the tree holds twenty-one suites, so a selector bug that escalates to most of them but not all stays green in every one of those tests. Partial escalation is the failure worth catching on exactly these paths; the threshold is blind to it and only sees total failure. Compare against the whole list instead, derived the way the script's full-suite branch derives it rather than pinned as a literal, so adding or disabling a suite does not need fourteen edits here. On a one-suite drop injected into that branch, thirteen of the converted tests go red; under the threshold all thirteen stayed green. The fourteenth is the cilium case, and equality shows it was never a full-suite path: packages/system/cilium is owned by cozystack.networking and resolves through the dependency graph, which reaches nineteen of the twenty-one suites. The two it misses, kuberture and securitygroup, sit on sources that are not *-application, and the source-to-suite mapping covers only those plus external-dns by name. Both controllers declare cozystack.networking as a dependency, so a CNI change that breaks either one runs neither suite. That gap belongs to the selector; the test is renamed for what it measures and pins the nineteen, so closing the gap fails it rather than passing unnoticed. Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three holes on the same seam: the script builds a list, never looks at whether the command that built it succeeded, and then decides from the empty result. The dependency indexes are built as `$(build_owners_index | sort -u)`, where the function is a single yq. A pipeline carries the status of its last command, so the script sees sort's zero and `set -e` never fires: a missing binary or a malformed source leaves the index empty, every changed path falls through to escalation, and CI runs the full Chainsaw suite on every pull request. The direction is safe and that is the problem — an empty index and "no owners matched" print the same thing, so the symptom is a full run, which reads as the conservatism the selector is supposed to apply. Selection can stay switched off indefinitely with nobody signalled. Check each half's status on its own and name yq once on stderr before taking the fallback the empty index would have reached anyway. The suite list one line up has the identical shape and a worse consequence: `$(find ... | sed | sort)` reports sort, never find, and errors went to /dev/null. That list is what every escalation prints. So a find that failed produced an empty list, and "escalate to the full suite" then emitted a blank line — which both e2e lanes read as "skip Chainsaw" before posting the required "E2E Tests" status green. The one outcome that means run everything delivered a run of nothing, on the pull request that earned a full run. Capture find on its own, check it, let its errors through. A failing find is not the only way to end up with an empty list, and the escalations are not its only reader, so the check sits once where the list is built. The examples/backups/<app>/ rule escalates nothing and still membership-tests against that list: with it empty the test fails, the app is not selected, and an edit to a backup harness that should run its suite reports nothing to run — output identical to a legitimately empty selection, on a path no escalation guard would ever see. An empty list is a broken enumeration, a moved directory or a wrong working directory, not a project without tests. Refuse to answer at all. Both lanes run the step under `bash -e`, so a non-zero exit fails the job rather than falling through to the empty selection this is here to prevent. Each is covered by a test that stubs the failing component and asserts both the exit status and that the script names what broke. Signed-off-by: Aleksei Sviridkin <f@lex.la>
Editing hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled selects the whole Chainsaw suite. The per-suite rule reads the suite name off the directory and ignores the suffix, so the change selects "backup"; the final intersection against the suites that actually exist finds no such suite and empties; and the safety net for an empty selection escalates to the full run. A disabled suite is registered nowhere and executed by nothing, which makes its edits the one file class that provably cannot regress a test. The mapping had it backwards: they bought the most expensive outcome the selector has. Rare in practice, and paid on every commit by anyone grooming or re-enabling such a suite. Classify *.disabled as inert ahead of the per-suite rule, and record it alongside the other rules that select nothing so the header stays an accurate enumeration. The safety net keeps its job for paths no rule has looked at, which is what it was added for. Signed-off-by: Aleksei Sviridkin <f@lex.la>
The helm unit sweep iterates packages/apps, core, extra, system and library, and never visits packages/tests. The cozy-lib-tests suite — the whole test coverage for a helper library 34 charts consume by symlink — runs only because packages/library/cozy-lib/Makefile carries a line delegating its own 'test' target to it. Deleting that one line takes the visited-package count for the suite from one to zero and the executions from one to zero, with the sweep still exiting 0 and still printing "All Helm unit tests passed": the no-suites backstop is satisfied by the other directories. Nothing else in the sweep could have caught it. Every check it makes — the Makefile test, the `make -n test` probe, the collection of failing directories — runs inside a directory the loop already reached, so a package that never enters the loop takes all of them with it. What is left is tests_found, and that counts suites across the whole run rather than per package, so the other directories keep it satisfied. Add the directory to the loop. It holds one package today, so the sweep grows from 76 executed suites to 77, and that suite costs between 0.1s and 0.3s over six runs against a sweep of roughly 31 seconds. The delegation stays for the ergonomics of `make -C packages/library/cozy-lib test`, which means cozy-lib-tests now runs twice; the point is that neither run is load-bearing alone. Covered by a new bats file: one test that the sweep reaches the directory and executes what it finds there, one that a red suite there still fails the sweep rather than being swallowed. Signed-off-by: Aleksei Sviridkin <f@lex.la>
📝 WalkthroughWalkthroughThe E2E selector now handles disabled suites, suite discovery failures, dependency-graph failures, and empty selections explicitly. Its tests use exact suite assertions. The Helm unit-test sweep now includes ChangesE2E suite selection
Helm unit-test sweep
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related issues
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/helm-unit-tests.bats`:
- Around line 62-66: Update the synthetic Makefile recipe in the test around
SWEEP to emit a distinctive marker before failing, capture SWEEP’s output while
preserving its exit code, and assert that both the packages/tests path and
marker appear in the output before checking for a nonzero status. This must
distinguish execution of the failing target from the no-suite guard.
In `@hack/select-e2e_test.bats`:
- Line 113: Remove the EXIT trap near the test setup in
hack/select-e2e_test.bats, and retain or add explicit rm -rf "$tmp" cleanup at
the end of the test body. Follow the no-EXIT-traps convention for hack/*.bats
tests while preserving temporary-directory cleanup.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: cf28e98b-cb03-44ab-8315-4d2cc96f79ff
📒 Files selected for processing (7)
.github/workflows/e2e-fork.yaml.github/workflows/pull-requests.yamldocs/agents/e2e-testing.mdhack/helm-unit-tests.batshack/helm-unit-tests.shhack/select-e2e.shhack/select-e2e_test.bats
| tmp=$(mktemp -d) | ||
| mkdir -p "$tmp/packages/tests/cozy-lib-tests" | ||
| printf 'test:\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile" | ||
| rc=0 | ||
| (cd "$tmp" && "$SWEEP" >/dev/null 2>&1) || rc=$? |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make this test prove that the failing packages/tests target ran.
The synthetic tree contains no other suites. If the sweep skips packages/tests, its no-suite guard can still produce a nonzero status. The test discards all output, so it cannot distinguish that case from a propagated make failure.
Add a marker to the failing recipe, capture the output with the exit code, and assert the package path and marker before asserting the nonzero status.
Proposed test adjustment
- printf 'test:\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile"
+ printf 'test:\n\t@echo SWEEP_REACHED_TESTS\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile"
rc=0
- (cd "$tmp" && "$SWEEP" >/dev/null 2>&1) || rc=$?
+ output=$(cd "$tmp" && "$SWEEP" 2>&1) || rc=$?
+ printf '%s\n' "$output" | grep -q "Running tests in packages/tests/cozy-lib-tests"
+ printf '%s\n' "$output" | grep -q SWEEP_REACHED_TESTS📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| tmp=$(mktemp -d) | |
| mkdir -p "$tmp/packages/tests/cozy-lib-tests" | |
| printf 'test:\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile" | |
| rc=0 | |
| (cd "$tmp" && "$SWEEP" >/dev/null 2>&1) || rc=$? | |
| tmp=$(mktemp -d) | |
| mkdir -p "$tmp/packages/tests/cozy-lib-tests" | |
| printf 'test:\n\t@echo SWEEP_REACHED_TESTS\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile" | |
| rc=0 | |
| output=$(cd "$tmp" && "$SWEEP" 2>&1) || rc=$? | |
| printf '%s\n' "$output" | grep -q "Running tests in packages/tests/cozy-lib-tests" | |
| printf '%s\n' "$output" | grep -q SWEEP_REACHED_TESTS |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/helm-unit-tests.bats` around lines 62 - 66, Update the synthetic
Makefile recipe in the test around SWEEP to emit a distinctive marker before
failing, capture SWEEP’s output while preserving its exit code, and assert that
both the packages/tests path and marker appear in the output before checking for
a nonzero status. This must distinguish execution of the failing target from the
no-suite guard.
| # today's selector, not a typo. The previous `wc -w -gt 5` assert passed on | ||
| # 19 and said nothing. | ||
| tmp=$(mktemp -d) | ||
| trap 'rm -rf "$tmp"' EXIT |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Remove the test-level EXIT trap.
Line 113 replaces Bats bookkeeping when this test fails. Bats can then omit the TAP failure line and end with an executed-test-count warning. Keep the explicit cleanup at the end of the test body instead. The convention in docs/agents/e2e-testing.md §3 forbids EXIT traps in hack/*.bats tests.
Proposed fix
- trap 'rm -rf "$tmp"' EXIT📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| trap 'rm -rf "$tmp"' EXIT |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/select-e2e_test.bats` at line 113, Remove the EXIT trap near the test
setup in hack/select-e2e_test.bats, and retain or add explicit rm -rf "$tmp"
cleanup at the end of the test body. Follow the no-EXIT-traps convention for
hack/*.bats tests while preserving temporary-directory cleanup.
What this PR does
Four fixes in e2e test selection and the helm unit sweep. Relates to #3650 #3652 #3655 #3646.
select-e2e built both dependency indexes as
$(build_owners_index | sort -u), so it read sort's exit status and never yq's (#3650). A broken yq left the index empty, every path fell through to escalation, and CI ran the full suite on every PR. The output was identical to "no owners matched", so it looked like the conservatism the selector is supposed to apply.The suite list one line up had the same shape and a worse result.
$(find ... | sed | sort)reports sort, never find, and errors went to /dev/null. Every escalation prints that list, so a failed find made "escalate to the full suite" emit a blank line, which both e2e lanes read as "skip Chainsaw" before postingE2E Testsgreen. The outcome that means run everything delivered a run of nothing.find is not the only way to get an empty list, and the escalations are not its only reader, so the check sits where the list is built. The
examples/backups/<app>/rule escalates nothing and still membership-tests against that list: with it empty, an edit to a backup harness that should run its suite reports nothing to run, on a path no escalation-side guard would ever see. An empty list is a broken enumeration, so the script refuses to answer at all. Both lanes run the step underbash -e, so the non-zero exit fails the job. I checked both workflows: neither overridesshell:, both capture the selector as a plain assignment.Fourteen tests used
[ "$(echo "$output" | wc -w)" -gt 5 ]to mean "everything was selected" (#3652). There are 21 suites, so that passes on any six, and partial escalation stayed green. They compare against the whole list now, derived the way the script's own full-suite branch derives it. Drop one suite from that branch and thirteen of them go red. All thirteen passed under the threshold.The fourteenth is the cilium case, and equality shows it was never a full-suite path at all. It resolves through the dependency graph, which reaches 19 of the 21 suites. The two it misses are kuberture and securitygroup, whose sources are not
*-application, and the mapping covers only those plus external-dns by name. Both controllers declarecozystack.networking, so a CNI change that breaks either one runs neither suite. That gap is #3665. The test is renamed for what it measures and pins the 19, so closing #3665 turns it red at the right moment.Editing
hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabledselected the full suite (#3655). The suffix was ignored when deriving the suite name, the intersection against existing suites emptied, and the empty-selection safety net escalated. A disabled suite runs nowhere, so its edits are the one file class that cannot regress a test, and they were buying the most expensive outcome the selector has.The selector now has a fourth outcome, a failed step rather than a selection, so the contract enumerations in the script header, in
docs/agents/e2e-testing.mdand in both workflow comments say so.The helm unit sweep never visited
packages/tests(#3646). cozy-lib-tests, the whole test coverage for a helper library that 34 charts consume by symlink, ran only becausepackages/library/cozy-lib/Makefiledelegates itstesttarget to it. Remove that one line and the suite goes from one execution to zero with the sweep still exiting 0 and still printing "All Helm unit tests passed". Every check the sweep makes runs inside a directory the loop already reached, so a package that never enters the loop takes all of them with it, and what is left istests_found, which counts suites across the whole run rather than per package. The directory is on the list now: 76 executed suites become 77, at 0.1s to 0.3s over six runs against a sweep of roughly 31 seconds. The delegation stays formake -C packages/library/cozy-lib testergonomics, so the suite runs twice. Neither run is load-bearing alone.Testing
hack/select-e2e_test.batsgoes from 27 tests to 32,hack/helm-unit-tests.batsis new. Each commit is green on its own: 27, 30, 32, 32.The new tests stub the failing component and assert the exit status plus the line naming what broke: a broken yq, a broken find, an empty suite list across all three diff classes, the disabled-suite path, and a guard that the disabled rule drops only that path and not the rest of the diff. The yq test asserts exit 0 on purpose. A missing dependency graph still has a correct answer, the full suite, and a missing suite list does not. Pinning both halves keeps the header and the docs from drifting into claiming otherwise.
Two suites fail on my macOS box and fail identically at
origin/main, so neither comes from this branch:hack/migration-seaweedfs-db-adopt.bats, where a container running as the host UID cannot write the bind-mounted temp dir under Colima, andhack/seaweedfs-naming-audit.bats, which I confirmed red at the merge base but did not dig into. Every otherhack/*.batswas run individually and is green.make unit-testsstops at the first failing file and both of those sort ahead ofselect-e2e_test.bats, so on macOS the suites this PR touches have to be run directly.Named improvements, not done here
.disabledrule keys on the file rather than the parked directory, so it is both wider and narrower than the property it wants. A support file underhack/e2e-chainsaw/backup/still buys a full run, and a.disabledfile dropped inside a live suite directory now selects nothing where it used to select that suite. The property covering both is one level up: a suite directory holding no livechainsaw-test.yamlis inert, testable with the samegrep -Fxqmembership check theexamples/backups/*/*arm already uses. It is not a free win, since a half-written suite directory escalates today and would then select nothing. A cheaper half step is narrowing the glob tohack/e2e-chainsaw/*/chainsaw-test.yaml*.disabled, because case globs cross/and the shipped pattern also matches nested paths.packages/tests/**is not classified by the selector and reaches the full suite through the unclassified fall-through instead of a rule. Tracked in ci: select-e2e escalates packages/tests edits by fall-through, printing a false 'unclassified path' call-to-action #3667.full_suite_list()in the bats file cannot fail. cozytest.sh's awk rewrites a lone}intoreturn 0; }, so a broken find inside the helper would return an empty expected set instead of a red test. Nothing false can pass today, because the script hard-fails on an empty suite list before any assert runs, but that helper is what every equality assert measures against.docs/x.mddiff returns 21 suites at head against 0 at the merge base. Only reachable in an already broken state, but the graph is consulted at the component lookup and in the closure walk, so moving the check to first use would charge nothing to docs, dashboards and inert-meta diffs.Screenshots
No UI changes.
Downstream repositories
Walked the trigger map in
docs/agents/contributing.mdfile by file against the diff: four files underhack/, one underdocs/agents/, and comments in the two e2e workflows. Nothing underhack/was moved or renamed and no make-target contract changed, so the ccp entry does not fire. It gates onhack/package.mkandhack/common-envs.mk, neither of which is in the diff. The node prerequisites inhack/e2e-prepare-cluster.batsthat ansible-cozystack mirrors by hand are untouched, and so is.github/workflows/tags.yaml.docs/agents/is internal contributor documentation, not a website source. No package was added, renamed or removed, and novalues.schema.jsonor version enum changed, so the website and provider entries do not fire either.Release note
Summary by CodeRabbit
Bug Fixes
packages/tests.Tests
Documentation