test(e2e): add chainsaw-native release upgrade testing lane - #3276

Code Review

This pull request introduces a release upgrade E2E testing lane to verify platform upgrades from the previous latest minor stable release to the current version. It adds composite GitHub Actions for sandbox lifecycle management, BATS orchestration scripts, and Kyverno Chainsaw suites for seeding and verifying workloads (MariaDB, PostgreSQL, Redis, VMs, and tenant Kubernetes clusters). The review feedback highlights a missing seeding file for PostgreSQL (upgrade-seed-postgres), recommends adding the --fail (-f) flag to curl when downloading assets to ensure proper error handling, suggests quoting the SANDBOX_NAME variable in shell commands to prevent word splitting, and proposes simplifying heavily escaped inline scripts in the upgrade BATS tests.

Actionable comments posted: 2

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/actions/e2e-prepare/action.yaml:
- Around line 28-35: Update the sandbox-name step around the shell variable key
to receive sandbox-suffix through the step’s env configuration, then reference
that environment variable in the shell condition and key extension instead of
directly interpolating inputs.sandbox-suffix. Preserve the existing default
behavior when the suffix is empty and the current SANDBOX_NAME and sandbox-name
outputs.

In `@hack/e2e-chainsaw-upgrade/verify/platform/chainsaw-test.yaml`:
- Around line 41-55: Validate that the baseline files exist before running
either comm comparison in the PersistentVolume and CrashLoopBackOff checks. Add
fail-fast guards for $dir/unbound-pv.txt and $dir/crashloop.txt, exiting nonzero
with an appropriate error when either is missing, while preserving the existing
PV failure gate and warning-only CrashLoopBackOff behavior.

---

Nitpick comments:
In `@hack/e2e-chainsaw-upgrade/seed/tenant-k8s/chainsaw-test.yaml`:
- Around line 20-22: Update the catch block in the tenant-k8s Chainsaw test to
retain the existing events diagnostics and add scoped describe diagnostics for
the KamajiControlPlane, TenantControlPlane, and MachineDeployment resources,
matching the diagnostic structure used by the sibling seed suites.

Actionable comments posted: 1

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/actions/e2e-prepare/action.yaml:
- Around line 3-8: Clarify the retry description in the composite action
documentation around the Talos e2e sandbox provisioning: state that the command
permits up to 3 total attempts, meaning 2 retries, and update the referenced
e2e-testing guidance citation or wording so both describe the same retry budget.

Verdict

LGTM with non-blocking notes

test(e2e): add chainsaw-native release upgrade testing lane (+1692/-90, 29 files). No CRITICAL/MAJOR defects after executing the available offline reproductions (helm template vs merge-base, dash/shellcheck on both new scripts, mutation-testing the new bats suite, source-level verification of the GitHub Actions / Chainsaw assumptions).

Findings

[MINOR] hack/upgrade-prev-version_test.bats:87-95 (guards hack/upgrade-prev-version.sh:41) — the test named "line match is dot-anchored (1.5 does not match v155.x)" is theatre. Removing the dot-escaping (sed 's/\./\\./g') still leaves all 10 tests green, because stable_desc's upstream filter already normalizes tag shapes so escaped vs unescaped patterns are equivalent (confirmed via a ~30k-combination differential check). A second mutation on the walk-down loop did correctly turn a different test red, so the rest of the suite is non-vacuous — only this one named test overclaims what it verifies.

Claim mismatches (non-blocking)

• [PARTIAL] PR body "Status" section is stale: commit 2c6b9140 already fixes the canary DB access path the body says "still needs a dev-cluster run".
• [UNVERIFIABLE] actionlint / live CI green — outside the hermetic toolset; the unit-tests portion was independently confirmed by re-running the bats suite.

Caveats (verification limits, not defects)

• Phase 5c corner-render for the new migrations.etcdAdoptSkipBackup toggle could not be executed: helm template packages/core/platform fails offline on a pre-existing unconditional lookup+fail in repository.yaml (reproduces identically on merge-base, not touched by this PR), and migration-hook.yaml is itself lookup-gated. Residual risk judged low — the consumed ETCD_ADOPT_SKIP_BACKUP env var is pre-existing, already unit-tested, and the wiring is a single unconditional-else block with no cross-reference or dependsOn.
• The chart_lint render_error + 2 missing_refs flagged by tooling are all confirmed pre-existing and unrelated (reproduced on merge-base; referencing templates untouched).
• Both new #!/bin/sh scripts pass shell-portability checks (shellcheck --shell=sh, dash -n, no bashisms, invoked only via shebang).
• Tenant-Kubernetes seed/verify suites are correctly gated off by default (UPGRADE_E2E_TENANT_K8S); zero risk to default CI.

NOT LGTM. The head commit rewrites two workflow gate expressions that an existing contract test pins verbatim, without updating that test. The unit-test job goes red, which skips finalize, which skips both the required E2E check and this PR's own new lane, so at this head nothing in the verification pipeline actually ran.

Business context: upgrade coverage is entirely absent today (there is no upgrade job anywhere in pull-requests.yaml on main), so a defect that only appears when an existing cluster moves to a new release ships unnoticed. This adds a lane that installs the previous stable release, seeds workloads with canary data, upgrades to the build under test, and asserts survival.

Blockers

B1: broadening the label gate breaks the contract test that guards it, which takes down the whole pipeline

File: .github/workflows/pull-requests.yaml:32 and :454

Issue: hack/promote-gate-contract.bats:122 and :126 pin these two gate expressions as exact string literals. This PR broadens both from github.event.label.name == 'full-e2e' to contains(fromJSON('["full-e2e","upgrade-e2e"]'), github.event.label.name) and does not update the test, so both grep -cF counts fall to zero.

Evidence: reproduced locally at f6c0ed4, running that contract file on its own prints Test failed: release PR E2E is a working manual full-e2e label opt-in (exit 1). The CI job log ends with the same sequence, + count=0, + [ 0 -eq 1 ], then make: *** [Makefile:165: bats-unit-tests] Error 1. In the run at this head, Unit & controller tests is failure and Finalize, E2E Tests and Upgrade E2E Test are all skipped.

Impact: the required E2E Tests check does not run on merit, it reports skipped. The new lane does not run either, even though the upgrade-e2e label is applied, so the commit titled "make the upgrade-e2e label opt-in actually start a run" currently stops every run instead. It is also why the lane's results are only visible in the run from nine days ago rather than at this head.

Fix: update both assertions to match the broadened expressions and add one asserting upgrade-e2e is in the allow-list, so the contract keeps guarding the gate rather than being routed around it.

On the lane's own red result

I checked this rather than taking it on trust, because a brand new upgrade lane failing on its first run is equally consistent with the lane being wrong. Two of the three failures carry a terminal mechanism and are platform defects, not lane defects.

The seaweedfs pair fails during template rendering: execution error at (seaweedfs/templates/seaweedfs.yaml:191:6) and (cozy-seaweedfs/templates/naming-guard.yaml:151:6), the naming guard refusing the release because volume names are derived from the release name. That is only reachable by upgrading an existing install, which is exactly the gap this lane closes. tenant-root/tenant-root then fails waiting on it, so that is a consequence rather than a fourth defect.

opensearch-operator fails with Deployment.apps "opensearch-operator-controller-manager" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is Recreate. That is the server-side-apply residue case, where the field set by the previous version survives into the new object. Upgrade-only again, and opensearch-rd is a downstream dependency of it, not a separate defect.

The third, vm-instance-test timing out on VirtualMachine/tenant-test/vm-instance-test status: 'InProgress', is a real observed regression signal, but the log does not establish a root cause, so I would not put it in the same category as the other two yet. Worth saying that explicitly rather than counting three equally proven findings.

All three belong in their own changes, so none of them block this one. I also confirmed the lane is genuinely advisory rather than taking the description's word for it: branch protection on main requires only pre-commit and E2E Tests, and this job's check name is Upgrade E2E Test, which is neither.

Non-blocking follow-ups

1. A permanently red advisory check trains reviewers to ignore it, and then it stops working on the day it catches something new. Nothing here records which failures are currently expected, so once this merges a fourth, unrelated failure is indistinguishable from the three known ones. Landing the three fixes first, or recording the known-failing set somewhere the lane prints, would keep it honest.

2. hack/e2e-chainsaw-upgrade/verify/platform/chainsaw-test.yaml:44 and :55: comm -13 "$dir/unbound-pv.txt" ... || true turns a missing baseline file into an empty diff and therefore a pass, so a verify phase run without its seed phase reports healthy having compared nothing. The sibling suite already handles this correctly at verify/vm/chainsaw-test.yaml:47, which fails fast with [ -f "$before" ] || { ...; exit 1; }. Applying the same guard here would make the two consistent. This was raised in the automated review and is still open.

3. hack/upgrade-prev-version_test.bats:87: the case named "line match is dot-anchored (1.5 does not match v155.x)" does not test that. I removed the sed 's/\./\\./g' escaping at hack/upgrade-prev-version.sh:41 and all ten cases stayed green, because stable_desc already constrains tags to vX.Y.Z and that makes the escaped and unescaped patterns equivalent over the resulting set. The shipped code is correct, only the test overclaims. An earlier review reached the same conclusion. The rest of the suite is not vacuous: unanchoring the stable-tag regex, and starting the minor walk at min instead of min - 1, each turned a case red.

4. .github/actions/e2e-download-assets/action.yaml:50: curl -sSL ... -o _out/assets/nocloud-amd64.raw.xz has no --fail, so an auth failure or 404 gets written into the disk image while curl exits 0, surfacing much later as a confusing decompression error. The behaviour is unchanged from the code this was extracted from, but it now serves two jobs, and the release PR path is precisely the one this lane exists for. Also raised in the automated review and still open.

5. The description does not name the costs it accepts: a second full platform install per run on a 32 vCPU runner with a 180 minute ceiling, doubled exposure to install-phase flakiness, and six verify suites to maintain as those apps change. I think the trade is worth making, but it should be stated rather than left implicit. The "Status" section also still says "Draft" and "unit-tests green locally", neither of which holds now.

6. hack/e2e-install-cozystack.bats is also being changed by another open PR, in the worker image catalog region rather than the readiness gate this one extracts. They should not conflict; I am flagging it only so whoever merges second knows to look.

What I checked and found clean

No EXIT or RETURN traps in either new pipeline BATS file or in any of the twelve new Chainsaw suites. hack/e2e-wait-hr-ready.sh uses if kubectl wait ...; then rather than capturing output into a variable, so it avoids the failure mode where an assignment with command substitution under set -e kills the shell before the captured output is ever printed. I checked every command-substitution assignment this diff adds and none has that shape. hack/upgrade-prev-version_test.bats is correctly picked up by the unit target rather than filtered out as an e2e file, and it declares ten cases and executes ten. The tenant Kubernetes suites gate off through an early exit 0 in a single script step and print a visible skip line. The lane runs fixed seed and verify paths with no selector, so unlike the app suite it cannot silently select nothing and report success; the one way it can report a non-failure having run nothing is the skip path in B1. The seed suites really do reuse the existing app manifests, all five relative paths resolve. Documentation is in sync, this diff adds the matching section to the E2E conventions doc.

Second round on the same head, posted separately so the additions actually reach you. Two things are new since my earlier review: a second blocker on the stale file header in the mariadb seed manifest, and a corrected statement of the version-resolver defect (the failing input is a single-component target like v1, not a two-component one). Everything else below stands as reported.

NOT LGTM. The head commit rewrites two workflow gate expressions that an existing contract test pins verbatim, without updating that test. The unit-test job goes red, which skips finalize, which skips both the required E2E check and this PR's own new lane, so at this head nothing in the verification pipeline actually ran.

Business context: upgrade coverage is entirely absent today (there is no upgrade job anywhere in pull-requests.yaml on main), so a defect that only appears when an existing cluster moves to a new release ships unnoticed. This adds a lane that installs the previous stable release, seeds workloads with canary data, upgrades to the build under test, and asserts survival.

Blockers

B1: broadening the label gate breaks the contract test that guards it, which takes down the whole pipeline

File: .github/workflows/pull-requests.yaml:32 and :454

Issue: hack/promote-gate-contract.bats:122 and :126 pin these two gate expressions as exact string literals. This PR broadens both from github.event.label.name == 'full-e2e' to contains(fromJSON('["full-e2e","upgrade-e2e"]'), github.event.label.name) and does not update the test, so both grep -cF counts fall to zero.

Evidence: reproduced locally at f6c0ed4, running that contract file on its own prints Test failed: release PR E2E is a working manual full-e2e label opt-in (exit 1). The CI job log ends with the same sequence, + count=0, + [ 0 -eq 1 ], then make: *** [Makefile:165: bats-unit-tests] Error 1. In the run at this head, Unit & controller tests is failure and Finalize, E2E Tests and Upgrade E2E Test are all skipped.

Impact: the required E2E Tests check does not run on merit, it reports skipped. The new lane does not run either, even though the upgrade-e2e label is applied, so the commit titled "make the upgrade-e2e label opt-in actually start a run" currently stops every run instead. It is also why the lane's results are only visible in the run from nine days ago rather than at this head.

Fix: update both assertions to match the broadened expressions and add one asserting upgrade-e2e is in the allow-list, so the contract keeps guarding the gate rather than being routed around it.

On the lane's own red result

I checked this rather than taking it on trust, because a brand new upgrade lane failing on its first run is equally consistent with the lane being wrong. Two of the three failures carry a terminal mechanism and are platform defects, not lane defects.

The seaweedfs pair fails during template rendering: execution error at (seaweedfs/templates/seaweedfs.yaml:191:6) and (cozy-seaweedfs/templates/naming-guard.yaml:151:6), the naming guard refusing the release because volume names are derived from the release name. That is only reachable by upgrading an existing install, which is exactly the gap this lane closes. tenant-root/tenant-root then fails waiting on it, so that is a consequence rather than a fourth defect.

opensearch-operator fails with Deployment.apps "opensearch-operator-controller-manager" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy type is Recreate. That is the server-side-apply residue case, where the field set by the previous version survives into the new object. Upgrade-only again, and opensearch-rd is a downstream dependency of it, not a separate defect.

The third, vm-instance-test timing out on VirtualMachine/tenant-test/vm-instance-test status: 'InProgress', is a real observed regression signal, but the log does not establish a root cause, so I would not put it in the same category as the other two yet. Worth saying that explicitly rather than counting three equally proven findings.

All three belong in their own changes, so none of them block this one. I also confirmed the lane is genuinely advisory rather than taking the description's word for it: branch protection on main requires only pre-commit and E2E Tests, and this job's check name is Upgrade E2E Test, which is neither.

Non-blocking follow-ups

1. A permanently red advisory check trains reviewers to ignore it, and then it stops working on the day it catches something new. Nothing here records which failures are currently expected, so once this merges a fourth, unrelated failure is indistinguishable from the three known ones. Landing the three fixes first, or recording the known-failing set somewhere the lane prints, would keep it honest.

2. hack/e2e-chainsaw-upgrade/verify/platform/chainsaw-test.yaml:44 and :55: comm -13 "$dir/unbound-pv.txt" ... || true turns a missing baseline file into an empty diff and therefore a pass, so a verify phase run without its seed phase reports healthy having compared nothing. The sibling suite already handles this correctly at verify/vm/chainsaw-test.yaml:47, which fails fast with [ -f "$before" ] || { ...; exit 1; }. Applying the same guard here would make the two consistent. This was raised in the automated review and is still open.

3. hack/upgrade-prev-version_test.bats:87: the case named "line match is dot-anchored (1.5 does not match v155.x)" does not test that. I removed the sed 's/\./\\./g' escaping at hack/upgrade-prev-version.sh:41 and all ten cases stayed green, because stable_desc already constrains tags to vX.Y.Z and that makes the escaped and unescaped patterns equivalent over the resulting set. The shipped code is correct, only the test overclaims. An earlier review reached the same conclusion. The rest of the suite is not vacuous: unanchoring the stable-tag regex, and starting the minor walk at min instead of min - 1, each turned a case red.

4. .github/actions/e2e-download-assets/action.yaml:50: curl -sSL ... -o _out/assets/nocloud-amd64.raw.xz has no --fail, so an auth failure or 404 gets written into the disk image while curl exits 0, surfacing much later as a confusing decompression error. The behaviour is unchanged from the code this was extracted from, but it now serves two jobs, and the release PR path is precisely the one this lane exists for. Also raised in the automated review and still open.

5. The description does not name the costs it accepts: a second full platform install per run on a 32 vCPU runner with a 180 minute ceiling, doubled exposure to install-phase flakiness, and six verify suites to maintain as those apps change. I think the trade is worth making, but it should be stated rather than left implicit. The "Status" section also still says "Draft" and "unit-tests green locally", neither of which holds now.

6. hack/e2e-install-cozystack.bats is also being changed by another open PR, in the worker image catalog region rather than the readiness gate this one extracts. They should not conflict; I am flagging it only so whoever merges second knows to look.

What I checked and found clean

No EXIT or RETURN traps in either new pipeline BATS file or in any of the twelve new Chainsaw suites. hack/e2e-wait-hr-ready.sh uses if kubectl wait ...; then rather than capturing output into a variable, so it avoids the failure mode where an assignment with command substitution under set -e kills the shell before the captured output is ever printed. I checked every command-substitution assignment this diff adds and none has that shape. hack/upgrade-prev-version_test.bats is correctly picked up by the unit target rather than filtered out as an e2e file, and it declares ten cases and executes ten. The tenant Kubernetes suites gate off through an early exit 0 in a single script step and print a visible skip line. The lane runs fixed seed and verify paths with no selector, so unlike the app suite it cannot silently select nothing and report success; the one way it can report a non-failure having run nothing is the skip path in B1. The seed suites really do reuse the existing app manifests, all five relative paths resolve. Documentation is in sync, this diff adds the matching section to the E2E conventions doc.

NOT LGTM. Second round on the same head. Two things are new since my previous review, and the rest of that review stands unchanged: a second blocker, the file header in the MariaDB seed suite prescribing the connection method the same file documents as fatal, and a corrected version of the resolver follow-up, where the defect is real but my earlier framing of it was wrong. I also tightened the evidence on the original blocker.

Blockers

B1: broadening the label gate breaks the contract test that guards it, which takes down the whole pipeline

Unchanged from the previous round, restated because it still blocks. .github/workflows/pull-requests.yaml:32 and :454 broaden the gate from github.event.label.name == 'full-e2e' to contains(fromJSON('["full-e2e","upgrade-e2e"]'), github.event.label.name), while hack/promote-gate-contract.bats:122 and :126 pin the old literals. Both grep -cF counts fall to zero, bats-unit-tests fails, Unit & controller tests goes red, Finalize is skipped, and with it both E2E Tests and Upgrade E2E Test.

New evidence this round. This is not an artifact of the branch trailing main. The branch is zero commits behind main, and hack/promote-gate-contract.bats is byte-identical between this head and origin/main, where it still pairs with the narrow expressions at lines 31 and 453. The test and the workflow agree on main, and this PR is what puts them out of step. Attribution is unambiguous.

Fix: update both assertions to the broadened expressions and add one asserting upgrade-e2e is in the allow-list, so the contract keeps guarding the gate instead of being routed around.

B2: the MariaDB seed header prescribes the connection method the same file documents as fatal

hack/e2e-chainsaw-upgrade/seed/mariadb/chainsaw-test.yaml:5 says the canary is written "over 127.0.0.1 from inside the primary pod". Lines 23 and 24 of the same file say the opposite, with a mechanism attached: "connecting to 127.0.0.1 fails (server does not bind loopback) and its retry OOM-kills the nano server container; the primary Service is clean". The code at lines 65 and 70 uses -h mariadb-test-primary, matching the second comment and contradicting the header.

Commit f598c253 ("fix upgrade canary DB access, verified on a live cluster") records this same finding from a dev cluster and changed the code and the step comment, but left the file header on the pre-fix approach. So the header describes the method that commit replaced.

The header is the first thing anyone reads here. Someone simplifying the suite back to what it claims to do reproduces an OOM-kill of the server container, and that surfaces as a timeout that looks unrelated rather than as a connection error, which is the expensive kind to debug. Rewriting the header to match -h mariadb-test-primary is the whole fix. One line of prose, no test needed.

Non-blocking follow-ups

Items 4 and 8 are new this round. The rest are unchanged from the previous review.

1. A permanently red advisory check trains reviewers to ignore it, and then it stops working on the day it catches something new. Nothing here records which failures are currently expected, so once this merges a fourth, unrelated failure is indistinguishable from the three known ones. Landing the three fixes first, or recording the known-failing set somewhere the lane prints, would keep it honest.

2. hack/e2e-chainsaw-upgrade/verify/platform/chainsaw-test.yaml:44 and :55: comm -13 "$dir/unbound-pv.txt" ... || true turns a missing baseline file into an empty diff and so into a pass, and a verify phase run without its seed phase reports healthy having compared nothing. The sibling suites handle this correctly already, verify/vm/chainsaw-test.yaml:47 and verify/redis both fail fast with [ -f "$before" ] || { ...; exit 1; }. The same guard here would make the three consistent. Raised in the automated review and still open.

3. hack/upgrade-prev-version_test.bats:87: the case named "line match is dot-anchored (1.5 does not match v155.x)" does not test that. I removed the sed 's/\./\\./g' escaping at hack/upgrade-prev-version.sh:41 and all ten cases stayed green, because stable_desc already constrains tags to vX.Y.Z, which makes the escaped and unescaped patterns equivalent over that set. The shipped code is correct, only the test overclaims. An earlier review reached the same conclusion. The rest of the suite holds up: unanchoring the stable-tag regex, and starting the minor walk at min instead of min - 1, each turned a case red.

4. hack/upgrade-prev-version.sh:56: a single-component target is silently misparsed instead of rejected. min=$(printf '%s' "$ver" | cut -d. -f2) returns the whole string when there is no second field, so v1 gives maj=1 min=1, the numeric case guard accepts 1:1 as well-formed, and the walk resolves the 1.0 line. Confirmed against a synthetic tag set: v1 returns v1.0.9 with exit 0. Two- and three-component targets are fine, cut finds a second field in both. Unreachable from CI, since the workflow requires three components before it sets a target at all, but this is a standalone tool with a ten-case suite and one more fixture would pin it.

5. .github/actions/e2e-download-assets/action.yaml:50: curl -sSL ... -o _out/assets/nocloud-amd64.raw.xz has no --fail, so an auth failure or 404 gets written into the disk image while curl exits 0, surfacing later as a confusing decompression error. Behaviour is unchanged from the code this was extracted from, but it now serves two jobs, and the release PR path is the one this lane exists for. Raised in the automated review and still open.

6. The description does not name the costs it accepts: a second full platform install per run on a 32 vCPU runner with a 180 minute ceiling, doubled exposure to install-phase flakiness, and six verify suites to maintain as those apps change. I think the trade is worth making, but it should be stated. The "Status" section also still says "Draft" and "unit-tests green locally", neither of which holds now.

7. hack/e2e-install-cozystack.bats is also being changed by another open PR, in the worker image catalog region rather than the readiness gate this one extracts. They should not conflict. Flagging it so whoever merges second knows to look.

8. Both phases write chainsaw-report.xml into the same directory and the workflow always uploads it as upgrade-chainsaw-report, so when the lane fails during seed the artifact is the seed report published under the verify name. The step comment assumes verify ran last, true only on the passing path. Copying per phase, or naming the artifact by the phase that produced it, removes the ambiguity in the case where someone is actually reading it.

Pre-existing, not introduced here

template-injection is disabled tree-wide in .github/zizmor.yml, which is why the new composites' ${{ inputs.sandbox-name }} interpolations into run: pass the gate. The values are sha256 hex so there is no exploitable path in this diff, but the new composite actions enlarge the surface that audit will cover once it is re-enabled. Worth attaching to the existing hardening item. Separately, hack/e2e-chainsaw/mariadb/mariadb.yaml carries hardcoded fixture credentials that predate this PR, and the new suites re-type the password literal in two more places, which argues for sourcing it from the manifest.

The lane's own red result, and what I verified clean

Unchanged from the previous round. Short form: of the three failures the lane surfaced, the seaweedfs naming-guard render failure and the opensearch spec.strategy.rollingUpdate rejection are platform defects reachable only by upgrading an existing install, which is the gap this lane exists to close. The vm-instance-test timeout is a real signal whose root cause the log does not establish, so I do not rank it with the other two. None of the three blocks this PR. The lane is advisory for real: branch protection requires only pre-commit and E2E Tests, and this job's check is Upgrade E2E Test.

Also unchanged. No EXIT or RETURN traps anywhere in the new suites or pipeline BATS files. No command-substitution assignment in this diff has the shape where set -e kills the shell before the captured output is printed. The new unit BATS declares ten cases, executes ten, and is picked up by the unit target. The tenant Kubernetes suites gate off through an early exit 0 and print a visible skip line. The lane has no suite selector, so it cannot silently select nothing, and the only way it reports a non-failure having run nothing is the skip path in B1. All five reused app manifest paths resolve. The migration-stamp assertion is reachable under the isp-full variant the lane selects. The E2E conventions doc gained the matching section in this diff.