Skip to content

fix(ci): close the silent-green gaps in e2e selection and the helm unit sweep - #3670

Merged
Aleksei Sviridkin (lexfrei) merged 4 commits into
mainfrom
fix/selector-hardening
Aug 8, 2026
Merged

fix(ci): close the silent-green gaps in e2e selection and the helm unit sweep#3670
Aleksei Sviridkin (lexfrei) merged 4 commits into
mainfrom
fix/selector-hardening

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Four fixes in e2e test selection and the helm unit sweep. Relates to #3650 #3652 #3655 #3646.

select-e2e built both dependency indexes as $(build_owners_index | sort -u), so it read sort's exit status and never yq's (#3650). A broken yq left the index empty, every path fell through to escalation, and CI ran the full suite on every PR. The output was identical to "no owners matched", so it looked like the conservatism the selector is supposed to apply.

The suite list one line up had the same shape and a worse result. $(find ... | sed | sort) reports sort, never find, and errors went to /dev/null. Every escalation prints that list, so a failed find made "escalate to the full suite" emit a blank line, which both e2e lanes read as "skip Chainsaw" before posting E2E Tests green. The outcome that means run everything delivered a run of nothing.

find is not the only way to get an empty list, and the escalations are not its only reader, so the check sits where the list is built. The examples/backups/<app>/ rule escalates nothing and still membership-tests against that list: with it empty, an edit to a backup harness that should run its suite reports nothing to run, on a path no escalation-side guard would ever see. An empty list is a broken enumeration, so the script refuses to answer at all. Both lanes run the step under bash -e, so the non-zero exit fails the job. I checked both workflows: neither overrides shell:, both capture the selector as a plain assignment.

Fourteen tests used [ "$(echo "$output" | wc -w)" -gt 5 ] to mean "everything was selected" (#3652). There are 21 suites, so that passes on any six, and partial escalation stayed green. They compare against the whole list now, derived the way the script's own full-suite branch derives it. Drop one suite from that branch and thirteen of them go red. All thirteen passed under the threshold.

The fourteenth is the cilium case, and equality shows it was never a full-suite path at all. It resolves through the dependency graph, which reaches 19 of the 21 suites. The two it misses are kuberture and securitygroup, whose sources are not *-application, and the mapping covers only those plus external-dns by name. Both controllers declare cozystack.networking, so a CNI change that breaks either one runs neither suite. That gap is #3665. The test is renamed for what it measures and pins the 19, so closing #3665 turns it red at the right moment.

Editing hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled selected the full suite (#3655). The suffix was ignored when deriving the suite name, the intersection against existing suites emptied, and the empty-selection safety net escalated. A disabled suite runs nowhere, so its edits are the one file class that cannot regress a test, and they were buying the most expensive outcome the selector has.

The selector now has a fourth outcome, a failed step rather than a selection, so the contract enumerations in the script header, in docs/agents/e2e-testing.md and in both workflow comments say so.

The helm unit sweep never visited packages/tests (#3646). cozy-lib-tests, the whole test coverage for a helper library that 34 charts consume by symlink, ran only because packages/library/cozy-lib/Makefile delegates its test target to it. Remove that one line and the suite goes from one execution to zero with the sweep still exiting 0 and still printing "All Helm unit tests passed". Every check the sweep makes runs inside a directory the loop already reached, so a package that never enters the loop takes all of them with it, and what is left is tests_found, which counts suites across the whole run rather than per package. The directory is on the list now: 76 executed suites become 77, at 0.1s to 0.3s over six runs against a sweep of roughly 31 seconds. The delegation stays for make -C packages/library/cozy-lib test ergonomics, so the suite runs twice. Neither run is load-bearing alone.

Testing

hack/select-e2e_test.bats goes from 27 tests to 32, hack/helm-unit-tests.bats is new. Each commit is green on its own: 27, 30, 32, 32.

The new tests stub the failing component and assert the exit status plus the line naming what broke: a broken yq, a broken find, an empty suite list across all three diff classes, the disabled-suite path, and a guard that the disabled rule drops only that path and not the rest of the diff. The yq test asserts exit 0 on purpose. A missing dependency graph still has a correct answer, the full suite, and a missing suite list does not. Pinning both halves keeps the header and the docs from drifting into claiming otherwise.

Two suites fail on my macOS box and fail identically at origin/main, so neither comes from this branch: hack/migration-seaweedfs-db-adopt.bats, where a container running as the host UID cannot write the bind-mounted temp dir under Colima, and hack/seaweedfs-naming-audit.bats, which I confirmed red at the merge base but did not dig into. Every other hack/*.bats was run individually and is green. make unit-tests stops at the first failing file and both of those sort ahead of select-e2e_test.bats, so on macOS the suites this PR touches have to be run directly.

Named improvements, not done here

  • The .disabled rule keys on the file rather than the parked directory, so it is both wider and narrower than the property it wants. A support file under hack/e2e-chainsaw/backup/ still buys a full run, and a .disabled file dropped inside a live suite directory now selects nothing where it used to select that suite. The property covering both is one level up: a suite directory holding no live chainsaw-test.yaml is inert, testable with the same grep -Fxq membership check the examples/backups/*/* arm already uses. It is not a free win, since a half-written suite directory escalates today and would then select nothing. A cheaper half step is narrowing the glob to hack/e2e-chainsaw/*/chainsaw-test.yaml*.disabled, because case globs cross / and the shipped pattern also matches nested paths.
  • packages/tests/** is not classified by the selector and reaches the full suite through the unclassified fall-through instead of a rule. Tracked in ci: select-e2e escalates packages/tests edits by fall-through, printing a false 'unclassified path' call-to-action #3667.
  • full_suite_list() in the bats file cannot fail. cozytest.sh's awk rewrites a lone } into return 0; }, so a broken find inside the helper would return an empty expected set instead of a red test. Nothing false can pass today, because the script hard-fails on an empty suite list before any assert runs, but that helper is what every equality assert measures against.
  • The yq check runs before the changed-file list is read, so with a broken graph even a docs-only diff escalates. Measured: a broken yq plus a docs/x.md diff returns 21 suites at head against 0 at the merge base. Only reachable in an already broken state, but the graph is consulted at the component lookup and in the closure walk, so moving the check to first use would charge nothing to docs, dashboards and inert-meta diffs.

Screenshots

No UI changes.

Downstream repositories

Walked the trigger map in docs/agents/contributing.md file by file against the diff: four files under hack/, one under docs/agents/, and comments in the two e2e workflows. Nothing under hack/ was moved or renamed and no make-target contract changed, so the ccp entry does not fire. It gates on hack/package.mk and hack/common-envs.mk, neither of which is in the diff. The node prerequisites in hack/e2e-prepare-cluster.bats that ansible-cozystack mirrors by hand are untouched, and so is .github/workflows/tags.yaml. docs/agents/ is internal contributor documentation, not a website source. No package was added, renamed or removed, and no values.schema.json or version enum changed, so the website and provider entries do not fire either.

Release note

fix(ci): select-e2e no longer reports an empty selection when it cannot enumerate the Chainsaw suites, which made an escalation to the full suite indistinguishable from "skip Chainsaw" and left the E2E gate green with nothing run. A failure to list the suites is now named on stderr and fails the step, a yq failure is named and still escalates to the full suite, and an edit to a disabled Chainsaw suite is inert instead of escalating to a full run.

Summary by CodeRabbit

  • Bug Fixes

    • Improved end-to-end test selection for disabled, empty, or partially changed test suites.
    • Test selection now safely escalates to the full suite when suite discovery or dependency analysis fails.
    • Helm unit-test sweeps now include tests under packages/tests.
  • Tests

    • Added coverage for suite discovery failures, disabled suites, command failures, and exact test selection.
    • Added Helm test-sweep coverage, including failure handling.
  • Documentation

    • Clarified test-impact analysis rules and suite-selection outcomes.

Fourteen tests read `[ "$(echo "$output" | wc -w)" -gt 5 ]` to mean "the
full suite was selected". A word count clears that bar on any selection of
six or more, and the tree holds twenty-one suites, so a selector bug that
escalates to most of them but not all stays green in every one of those
tests. Partial escalation is the failure worth catching on exactly these
paths; the threshold is blind to it and only sees total failure.

Compare against the whole list instead, derived the way the script's
full-suite branch derives it rather than pinned as a literal, so adding or
disabling a suite does not need fourteen edits here. On a one-suite drop
injected into that branch, thirteen of the converted tests go red; under
the threshold all thirteen stayed green.

The fourteenth is the cilium case, and equality shows it was never a
full-suite path: packages/system/cilium is owned by cozystack.networking
and resolves through the dependency graph, which reaches nineteen of the
twenty-one suites. The two it misses, kuberture and securitygroup, sit on
sources that are not *-application, and the source-to-suite mapping covers
only those plus external-dns by name. Both controllers declare
cozystack.networking as a dependency, so a CNI change that breaks either
one runs neither suite. That gap belongs to the selector; the test is
renamed for what it measures and pins the nineteen, so closing the gap
fails it rather than passing unnoticed.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Three holes on the same seam: the script builds a list, never looks at
whether the command that built it succeeded, and then decides from the
empty result.

The dependency indexes are built as `$(build_owners_index | sort -u)`,
where the function is a single yq. A pipeline carries the status of its
last command, so the script sees sort's zero and `set -e` never fires: a
missing binary or a malformed source leaves the index empty, every changed
path falls through to escalation, and CI runs the full Chainsaw suite on
every pull request. The direction is safe and that is the problem — an
empty index and "no owners matched" print the same thing, so the symptom
is a full run, which reads as the conservatism the selector is supposed to
apply. Selection can stay switched off indefinitely with nobody signalled.
Check each half's status on its own and name yq once on stderr before
taking the fallback the empty index would have reached anyway.

The suite list one line up has the identical shape and a worse
consequence: `$(find ... | sed | sort)` reports sort, never find, and
errors went to /dev/null. That list is what every escalation prints. So a
find that failed produced an empty list, and "escalate to the full suite"
then emitted a blank line — which both e2e lanes read as "skip Chainsaw"
before posting the required "E2E Tests" status green. The one outcome that
means run everything delivered a run of nothing, on the pull request that
earned a full run. Capture find on its own, check it, let its errors
through.

A failing find is not the only way to end up with an empty list, and the
escalations are not its only reader, so the check sits once where the list
is built. The examples/backups/<app>/ rule escalates nothing and still
membership-tests against that list: with it empty the test fails, the app
is not selected, and an edit to a backup harness that should run its suite
reports nothing to run — output identical to a legitimately empty
selection, on a path no escalation guard would ever see. An empty list is
a broken enumeration, a moved directory or a wrong working directory, not
a project without tests. Refuse to answer at all. Both lanes run the step
under `bash -e`, so a non-zero exit fails the job rather than falling
through to the empty selection this is here to prevent.

Each is covered by a test that stubs the failing component and asserts
both the exit status and that the script names what broke.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Editing hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled selects the
whole Chainsaw suite. The per-suite rule reads the suite name off the
directory and ignores the suffix, so the change selects "backup"; the
final intersection against the suites that actually exist finds no such
suite and empties; and the safety net for an empty selection escalates to
the full run.

A disabled suite is registered nowhere and executed by nothing, which
makes its edits the one file class that provably cannot regress a test.
The mapping had it backwards: they bought the most expensive outcome the
selector has. Rare in practice, and paid on every commit by anyone
grooming or re-enabling such a suite.

Classify *.disabled as inert ahead of the per-suite rule, and record it
alongside the other rules that select nothing so the header stays an
accurate enumeration. The safety net keeps its job for paths no rule has
looked at, which is what it was added for.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
The helm unit sweep iterates packages/apps, core, extra, system and
library, and never visits packages/tests. The cozy-lib-tests suite — the
whole test coverage for a helper library 34 charts consume by symlink —
runs only because packages/library/cozy-lib/Makefile carries a line
delegating its own 'test' target to it. Deleting that one line takes the
visited-package count for the suite from one to zero and the executions
from one to zero, with the sweep still exiting 0 and still printing "All
Helm unit tests passed": the no-suites backstop is satisfied by the other
directories.

Nothing else in the sweep could have caught it. Every check it makes — the
Makefile test, the `make -n test` probe, the collection of failing
directories — runs inside a directory the loop already reached, so a
package that never enters the loop takes all of them with it. What is left
is tests_found, and that counts suites across the whole run rather than
per package, so the other directories keep it satisfied.

Add the directory to the loop. It holds one package today, so the sweep
grows from 76 executed suites to 77, and that suite costs between 0.1s and
0.3s over six runs against a sweep of roughly 31 seconds. The delegation stays for the ergonomics
of `make -C packages/library/cozy-lib test`, which means cozy-lib-tests
now runs twice; the point is that neither run is load-bearing alone.

Covered by a new bats file: one test that the sweep reaches the directory
and executes what it finds there, one that a red suite there still fails
the sweep rather than being swallowed.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
@github-actions github-actions Bot added area/ci Issues or PRs related to CI workflows, GitHub Actions, automation size/L This PR changes 100-499 lines, ignoring generated files kind/bug Categorizes issue or PR as related to a bug labels Aug 8, 2026
@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The E2E selector now handles disabled suites, suite discovery failures, dependency-graph failures, and empty selections explicitly. Its tests use exact suite assertions. The Helm unit-test sweep now includes packages/tests with execution and failure-propagation tests.

Changes

E2E suite selection

Layer / File(s) Summary
Selector classification and fallback
.github/workflows/*.yaml, docs/agents/e2e-testing.md, hack/select-e2e.sh
The selector excludes disabled suites, rejects missing or empty suite listings, detects yq failures, and centralizes full-suite escalation. Documentation describes the resulting classifications.
Selector test coverage
hack/select-e2e_test.bats
Tests now assert exact suite sets and cover disabled suites, broken find and yq commands, empty enumeration, partial selections, and unclassified paths.

Helm unit-test sweep

Layer / File(s) Summary
packages/tests sweep integration
hack/helm-unit-tests.sh, hack/helm-unit-tests.bats
The sweep scans packages/tests. Tests verify cozy-lib-tests execution and nonzero propagation from a failing suite.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related issues

Possibly related PRs

Suggested labels: area/testing

Suggested reviewers: myasnikovdaniil

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main changes: hardening E2E selection and expanding the Helm unit-test sweep.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/selector-hardening

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/helm-unit-tests.bats`:
- Around line 62-66: Update the synthetic Makefile recipe in the test around
SWEEP to emit a distinctive marker before failing, capture SWEEP’s output while
preserving its exit code, and assert that both the packages/tests path and
marker appear in the output before checking for a nonzero status. This must
distinguish execution of the failing target from the no-suite guard.

In `@hack/select-e2e_test.bats`:
- Line 113: Remove the EXIT trap near the test setup in
hack/select-e2e_test.bats, and retain or add explicit rm -rf "$tmp" cleanup at
the end of the test body. Follow the no-EXIT-traps convention for hack/*.bats
tests while preserving temporary-directory cleanup.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: cf28e98b-cb03-44ab-8315-4d2cc96f79ff

📥 Commits

Reviewing files that changed from the base of the PR and between 92ec49e and 33bd161.

📒 Files selected for processing (7)
  • .github/workflows/e2e-fork.yaml
  • .github/workflows/pull-requests.yaml
  • docs/agents/e2e-testing.md
  • hack/helm-unit-tests.bats
  • hack/helm-unit-tests.sh
  • hack/select-e2e.sh
  • hack/select-e2e_test.bats

Comment thread hack/helm-unit-tests.bats
Comment on lines +62 to +66
tmp=$(mktemp -d)
mkdir -p "$tmp/packages/tests/cozy-lib-tests"
printf 'test:\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile"
rc=0
(cd "$tmp" && "$SWEEP" >/dev/null 2>&1) || rc=$?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Make this test prove that the failing packages/tests target ran.

The synthetic tree contains no other suites. If the sweep skips packages/tests, its no-suite guard can still produce a nonzero status. The test discards all output, so it cannot distinguish that case from a propagated make failure.

Add a marker to the failing recipe, capture the output with the exit code, and assert the package path and marker before asserting the nonzero status.

Proposed test adjustment
-    printf 'test:\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile"
+    printf 'test:\n\t@echo SWEEP_REACHED_TESTS\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile"
     rc=0
-    (cd "$tmp" && "$SWEEP" >/dev/null 2>&1) || rc=$?
+    output=$(cd "$tmp" && "$SWEEP" 2>&1) || rc=$?
+    printf '%s\n' "$output" | grep -q "Running tests in packages/tests/cozy-lib-tests"
+    printf '%s\n' "$output" | grep -q SWEEP_REACHED_TESTS
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
tmp=$(mktemp -d)
mkdir -p "$tmp/packages/tests/cozy-lib-tests"
printf 'test:\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile"
rc=0
(cd "$tmp" && "$SWEEP" >/dev/null 2>&1) || rc=$?
tmp=$(mktemp -d)
mkdir -p "$tmp/packages/tests/cozy-lib-tests"
printf 'test:\n\t@echo SWEEP_REACHED_TESTS\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile"
rc=0
output=$(cd "$tmp" && "$SWEEP" 2>&1) || rc=$?
printf '%s\n' "$output" | grep -q "Running tests in packages/tests/cozy-lib-tests"
printf '%s\n' "$output" | grep -q SWEEP_REACHED_TESTS
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/helm-unit-tests.bats` around lines 62 - 66, Update the synthetic
Makefile recipe in the test around SWEEP to emit a distinctive marker before
failing, capture SWEEP’s output while preserving its exit code, and assert that
both the packages/tests path and marker appear in the output before checking for
a nonzero status. This must distinguish execution of the failing target from the
no-suite guard.

Comment thread hack/select-e2e_test.bats
# today's selector, not a typo. The previous `wc -w -gt 5` assert passed on
# 19 and said nothing.
tmp=$(mktemp -d)
trap 'rm -rf "$tmp"' EXIT

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Remove the test-level EXIT trap.

Line 113 replaces Bats bookkeeping when this test fails. Bats can then omit the TAP failure line and end with an executed-test-count warning. Keep the explicit cleanup at the end of the test body instead. The convention in docs/agents/e2e-testing.md §3 forbids EXIT traps in hack/*.bats tests.

Proposed fix
-    trap 'rm -rf "$tmp"' EXIT
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
trap 'rm -rf "$tmp"' EXIT
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/select-e2e_test.bats` at line 113, Remove the EXIT trap near the test
setup in hack/select-e2e_test.bats, and retain or add explicit rm -rf "$tmp"
cleanup at the end of the test body. Follow the no-EXIT-traps convention for
hack/*.bats tests while preserving temporary-directory cleanup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci Issues or PRs related to CI workflows, GitHub Actions, automation kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant