[Backport release-1.6] fix(seaweedfs): make naming audit fail closed on kubectl and payload errors - #3474
Merged
Merged
Conversation
…errors hack/seaweedfs-naming-audit.sh was fail-open: any kubectl failure produced an empty table indistinguishable from an honestly clean fleet (namespace LIST failure = zero namespaces walked, secret/pvc/sts LIST failures = zero findings, all under 2>/dev/null with no error handling). The runbook uses this output as the gate before PVC deletion, as post-deletion verification, and as an upgrade precondition, so a transient API error could green-light destroying data (#3431). Route every kubectl call through a run_kubectl helper: non-zero exit prints a FATAL line naming the failed query and propagates the code up the whole chain (enumerations restructured to capture-then-check, since an exit inside $(...) dies with the subshell). By-name GETs distinguish legitimate absence from real errors via --ignore-not-found on a separate existence check; a Secret that exists but has no decodable release payload, or one that decodes without a chart name, is corrupt state and fails loudly instead of silently dropping the tenant. Incomplete PV-age evidence now marks the generation incomplete and falls to the safe no-direction branch, so a failed GET can no longer flip OVERLAP into a wrong Step-3 deletion candidate. A successful run with zero findings still prints the same bytes as before. Tests: 14 new cozytest cases — failure injection for every LIST and by-name GET site, corrupt-payload variants, and full-output golden diffs for the success paths (verified byte-identical to the pre-change script). 25/25 green. Fixes #3431 Assisted-By: Claude <noreply@anthropic.com> Assisted-By: GPT-5 <noreply@openai.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> (cherry picked from commit 3dc6cbd)
…odelled calls Review findings from #3436, plus what an adversarial re-check of the first attempt at them turned up. The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals "config" (the release`s values) after "chart", so a values subtree that spells chart.metadata.name shadowed the real chart name: the release then read as non-SeaweedFS, the tenant silently vanished from the report, and the script exited 0 -- the precise false clean this PR exists to prevent, reachable without any corruption at all. It now takes the FIRST match. The path stays adjacent on purpose: a looser "any name after metadata" matches chart.templates[].name, which Helm emits immediately after metadata on every healthy release, so the review`s suggested relaxation would have returned a template path for every tenant. Over-strictness fails loudly and recoverably; over-looseness deletes data quietly. Newlines are now folded before matching, so a pretty-printed payload parses at all rather than aborting, and whitespace around the punctuation is tolerated. first_deployed got the same treatment -- it had the identical greedy, single- line-only shape, and a spaced payload silently dropped the PV-vintage row from the report. The FATAL message names the path it read and the shape it expected, so a future Helm format change is diagnosable instead of looking like real corruption. Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy) each byte-compared against the same golden as the compact payload, so the shape must make no difference to the report. Mutation-checked -- restoring last-match extraction fails the decoy test on its own, restoring the single-line matcher fails the pretty test. The test fake answered any unmodelled kubectl invocation with exit 0 and empty stdout, the fail-open shape this script was rewritten to reject, and enough to let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97 naming the invocation. Verified inert first by instrumenting the fake: 88 invocations across the suite, none unmodelled. Also added the missing test for the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it. The runbook now tells the operator to read the exit code: a non-zero exit means the table is incomplete and no step may be taken on it. That contract lived only in the script header, while the runbook is what the operator follows. 29/29 tests green; sh -n clean. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> (cherry picked from commit e02adad)
This was referenced Aug 3, 2026
myasnikovdaniil
added a commit
that referenced
this pull request
Aug 4, 2026
## Why `release-1.6` was created at v1.6.0's merge commit on 2026-07-22, and the finalize fixes landed on `main` after that. Workflow files run from the ref they fire on, so a promote PR based on `release-1.6` runs **this** copy of `pull-requests-release.yaml`, not `main`'s. Shipping v1.6.1 from this line today would repeat two defects v1.6.0 already hit. ## What this carries Three upstream commits, after which this file is **byte-identical to `main`'s copy** — so later backports touching it will not conflict. | Upstream | Change | | --- | --- | | `01e1e7188` | `persist-credentials: false` on the checkout | | `f49d54a68` | publish the release with the merged changelog as its body | | `ba67fea7d` | drop `paths-ignore` from the trigger | ### persist-credentials Without it the checkout persists `GITHUB_TOKEN` as an `http.extraheader`, which silently wins over the app token injected by `git remote set-url`. The stable tag then pushes as `GITHUB_TOKEN`, which creates no workflow run, so `tags.yaml` never fires and its `generate-changelog` and `update-website-docs` backstops stay silent. That is exactly what happened to v1.6.0. ### Changelog as release body Without it the release publishes with the draft's own body, `"Promoted from vX.Y.Z-rc.N"`. **These two compound into a permanent defect on a maintenance line.** A patch's changelog is committed to `release-1.6.1`, merged into `release-1.6`, and never reaches `main` — and `update-releasenotes.yaml` only watches `main`, so nothing would ever sync it. The tag-time backstop that would have ported it is dead because of the credential bug. v1.6.1 would ship with placeholder release notes permanently, with no error anywhere. ### paths-ignore Latent rather than active. A promote PR carrying only `docs/changelogs/vX.Y.Z.md` would be dropped by the filter, producing no finalize run, no tag and no error. v1.6.1 will carry tag-string rewrites so it would not have fired, but the filter has no remaining purpose now that the promote PR always carries a changelog. ## On the cherry-pick policy `docs/release.md`'s skip rule says CI-only changes do not belong in a patch. That rule governs release *contents*; backporting release *machinery* so the line is releasable at all is a different thing, and there is precedent already on this branch — #3473, #3474 and #3514 are all CI/release backports. ## Verification `actionlint` clean. `zizmor` clean, no findings. YAML parses. No bats suite references this workflow, and `hack/promote-gate-contract.bats` does not exist on this line, so there is no stale contract test to trip. The diff against `origin/main`'s copy of the file is empty. ## Not included Three known gaps on this line are deliberately out of scope, none of which needs a code change to ship v1.6.1: - `e2e-tag.yaml` is absent and `tags.yaml` has no `rc-e2e` job, so no E2E runs on the rc. Handled by dispatching **E2E Release Tag from `main`** against the published rc tag, which also supplies the promote gate's alternate evidence. - `pull-requests.yaml` has no `labeled` trigger, so adding `full-e2e` to the promote PR starts nothing. Backporting that needs the label-event guards too, and rc-time validation makes it unnecessary here. - finalize is still the pre-#3456 monolith, so a mid-registry failure is not re-runnable and needs the documented hand recovery. ```release-note NONE ```
myasnikovdaniil
added a commit
that referenced
this pull request
Aug 19, 2026
v1.6.0's own upgrade notes tell operators to run hack/seaweedfs-naming-audit.sh, and the copy that shipped in v1.6.0 silences every kubectl call with 2>/dev/null. A timeout or an RBAC denial therefore prints an empty "all clean" table indistinguishable from a genuinely clean fleet, and that output gates a runbook step that deletes PVCs. Anyone following those notes today can still get a false clean. Verified per tag: v1.6.0 has 11 blanket redirections and no run_kubectl wrapper; v1.6.1, v1.6.2 and v1.5.4 have 4 and 18 respectively, the fail-closed shape from #3436 and its backport #3474. Published changelogs are historical records and are not normally edited. This one is an exception on purpose, because the instruction it carries is still live and still wrong. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
myasnikovdaniil
added a commit
that referenced
this pull request
Aug 19, 2026
…udit note (#3910) Adds the changelog for `v1.5.4`, and corrects one still-live instruction in v1.6.0's. The generated changelog is a good inventory and every one of its bullets is kept. What it had no layer for is the operator: no required-actions section, no runnable pre-upgrade checks, and the SeaweedFS 4.31 rename left as three isolated fix bullets that never say what to do about it. That is what is added on top, in the shape v1.6.0's changelog uses. Base widened to `v1.5.2`. v1.5.3 was tagged but its release was never published and its changelog PR was closed unmerged, so `docs/changelogs/v1.5.3.md` exists on no ref and a v1.5.3 base leaves #3212 (persistent EFI/TPM state) and #3194 (filer postgres2 connection pool) documented nowhere at all. Coverage is 82/82 commits in range. Three claims corrected. The slot 45 divergence is a skip rather than an ordering problem: v1.5.4 is the first 1.5.x release stamped `targetVersion: 46`, and `run-migrations.sh` loops `seq CURRENT (TARGET - 1)`, so a cluster that reaches 46 runs slots 46 through 53 on the way to 1.6 and never executes 1.6's own slot 45. Its chart-side half was missing too and is not redundant, because a fresh v1.5.4 install is stamped 46 having never run any slot, so only the chart can reach that population. And the S3 checksum bullet named a barman-cloud plugin path that does not exist on `release-1.5`. The audit re-run warning names v1.6.0 rather than an earlier release, because the script does not exist at v1.5.2 at all. v1.5.4 is the first 1.5.x release to carry it, and it carries the fail-closed version. The second commit edits `docs/changelogs/v1.6.0.md`, which is deliberate rather than an accident of scope. Published changelogs are historical records and are normally left alone, but v1.6.0's upgrade notes still tell operators to run `hack/seaweedfs-naming-audit.sh`, and the copy that shipped in v1.6.0 silences every `kubectl` call with `2>/dev/null`. A timeout or an RBAC denial therefore prints an empty all-clean table indistinguishable from a genuinely clean fleet, and that output gates a runbook step that deletes PVCs. Verified per tag: v1.6.0 carries 11 blanket redirections and no `run_kubectl` wrapper, while v1.6.1, v1.6.2 and v1.5.4 carry 4 and 18, the fail-closed shape from #3436 and its backport #3474.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #3436 to
release-1.6.