Skip to content

[Backport release-1.6] fix(seaweedfs): make naming audit fail closed on kubectl and payload errors - #3474

Merged
myasnikovdaniil merged 2 commits into
release-1.6from
backport-3436-to-release-1.6
Jul 28, 2026
Merged

[Backport release-1.6] fix(seaweedfs): make naming audit fail closed on kubectl and payload errors#3474
myasnikovdaniil merged 2 commits into
release-1.6from
backport-3436-to-release-1.6

Conversation

@github-actions

Copy link
Copy Markdown

Description

Backport of #3436 to release-1.6.

…errors

hack/seaweedfs-naming-audit.sh was fail-open: any kubectl failure
produced an empty table indistinguishable from an honestly clean fleet
(namespace LIST failure = zero namespaces walked, secret/pvc/sts LIST
failures = zero findings, all under 2>/dev/null with no error handling).
The runbook uses this output as the gate before PVC deletion, as
post-deletion verification, and as an upgrade precondition, so a
transient API error could green-light destroying data (#3431).

Route every kubectl call through a run_kubectl helper: non-zero exit
prints a FATAL line naming the failed query and propagates the code up
the whole chain (enumerations restructured to capture-then-check, since
an exit inside $(...) dies with the subshell). By-name GETs distinguish
legitimate absence from real errors via --ignore-not-found on a
separate existence check; a Secret that exists but has no decodable
release payload, or one that decodes without a chart name, is corrupt
state and fails loudly instead of silently dropping the tenant.
Incomplete PV-age evidence now marks the generation incomplete and
falls to the safe no-direction branch, so a failed GET can no longer
flip OVERLAP into a wrong Step-3 deletion candidate. A successful run
with zero findings still prints the same bytes as before.

Tests: 14 new cozytest cases — failure injection for every LIST and
by-name GET site, corrupt-payload variants, and full-output golden
diffs for the success paths (verified byte-identical to the pre-change
script). 25/25 green.

Fixes #3431

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 3dc6cbd)
…odelled calls

Review findings from #3436, plus what an adversarial re-check of the first
attempt at them turned up.

The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals
"config" (the release`s values) after "chart", so a values subtree that spells
chart.metadata.name shadowed the real chart name: the release then read as
non-SeaweedFS, the tenant silently vanished from the report, and the script
exited 0 -- the precise false clean this PR exists to prevent, reachable
without any corruption at all. It now takes the FIRST match. The path stays
adjacent on purpose: a looser "any name after metadata" matches
chart.templates[].name, which Helm emits immediately after metadata on every
healthy release, so the review`s suggested relaxation would have returned a
template path for every tenant. Over-strictness fails loudly and recoverably;
over-looseness deletes data quietly.

Newlines are now folded before matching, so a pretty-printed payload parses at
all rather than aborting, and whitespace around the punctuation is tolerated.
first_deployed got the same treatment -- it had the identical greedy, single-
line-only shape, and a spaced payload silently dropped the PV-vintage row from
the report.

The FATAL message names the path it read and the shape it expected, so a future
Helm format change is diagnosable instead of looking like real corruption.

Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy)
each byte-compared against the same golden as the compact payload, so the shape
must make no difference to the report. Mutation-checked -- restoring last-match
extraction fails the decoy test on its own, restoring the single-line matcher
fails the pretty test.

The test fake answered any unmodelled kubectl invocation with exit 0 and empty
stdout, the fail-open shape this script was rewritten to reject, and enough to
let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97
naming the invocation. Verified inert first by instrumenting the fake: 88
invocations across the suite, none unmodelled. Also added the missing test for
the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it.

The runbook now tells the operator to read the exit code: a non-zero exit means
the table is incomplete and no step may be taken on it. That contract lived only
in the script header, while the runbook is what the operator follows.

29/29 tests green; sh -n clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit e02adad)
@dosubot dosubot Bot added area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) area/testing Issues or PRs related to testing (e2e, bats, unit tests) backport Should change be backported on previous release kind/bug Categorizes issue or PR as related to a bug labels Jul 28, 2026
@myasnikovdaniil
myasnikovdaniil merged commit 9b1d744 into release-1.6 Jul 28, 2026
9 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-3436-to-release-1.6 branch July 28, 2026 12:24
myasnikovdaniil added a commit that referenced this pull request Aug 4, 2026
## Why

`release-1.6` was created at v1.6.0's merge commit on 2026-07-22, and
the finalize fixes landed on `main` after that. Workflow files run from
the ref they fire on, so a promote PR based on `release-1.6` runs
**this** copy of `pull-requests-release.yaml`, not `main`'s. Shipping
v1.6.1 from this line today would repeat two defects v1.6.0 already hit.

## What this carries

Three upstream commits, after which this file is **byte-identical to
`main`'s copy** — so later backports touching it will not conflict.

| Upstream | Change |
| --- | --- |
| `01e1e7188` | `persist-credentials: false` on the checkout |
| `f49d54a68` | publish the release with the merged changelog as its
body |
| `ba67fea7d` | drop `paths-ignore` from the trigger |

### persist-credentials

Without it the checkout persists `GITHUB_TOKEN` as an
`http.extraheader`, which silently wins over the app token injected by
`git remote set-url`. The stable tag then pushes as `GITHUB_TOKEN`,
which creates no workflow run, so `tags.yaml` never fires and its
`generate-changelog` and `update-website-docs` backstops stay silent.
That is exactly what happened to v1.6.0.

### Changelog as release body

Without it the release publishes with the draft's own body, `"Promoted
from vX.Y.Z-rc.N"`.

**These two compound into a permanent defect on a maintenance line.** A
patch's changelog is committed to `release-1.6.1`, merged into
`release-1.6`, and never reaches `main` — and `update-releasenotes.yaml`
only watches `main`, so nothing would ever sync it. The tag-time
backstop that would have ported it is dead because of the credential
bug. v1.6.1 would ship with placeholder release notes permanently, with
no error anywhere.

### paths-ignore

Latent rather than active. A promote PR carrying only
`docs/changelogs/vX.Y.Z.md` would be dropped by the filter, producing no
finalize run, no tag and no error. v1.6.1 will carry tag-string rewrites
so it would not have fired, but the filter has no remaining purpose now
that the promote PR always carries a changelog.

## On the cherry-pick policy

`docs/release.md`'s skip rule says CI-only changes do not belong in a
patch. That rule governs release *contents*; backporting release
*machinery* so the line is releasable at all is a different thing, and
there is precedent already on this branch — #3473, #3474 and #3514 are
all CI/release backports.

## Verification

`actionlint` clean. `zizmor` clean, no findings. YAML parses. No bats
suite references this workflow, and `hack/promote-gate-contract.bats`
does not exist on this line, so there is no stale contract test to trip.
The diff against `origin/main`'s copy of the file is empty.

## Not included

Three known gaps on this line are deliberately out of scope, none of
which needs a code change to ship v1.6.1:

- `e2e-tag.yaml` is absent and `tags.yaml` has no `rc-e2e` job, so no
E2E runs on the rc. Handled by dispatching **E2E Release Tag from
`main`** against the published rc tag, which also supplies the promote
gate's alternate evidence.
- `pull-requests.yaml` has no `labeled` trigger, so adding `full-e2e` to
the promote PR starts nothing. Backporting that needs the label-event
guards too, and rc-time validation makes it unnecessary here.
- finalize is still the pre-#3456 monolith, so a mid-registry failure is
not re-runnable and needs the documented hand recovery.

```release-note
NONE
```
myasnikovdaniil added a commit that referenced this pull request Aug 19, 2026
v1.6.0's own upgrade notes tell operators to run
hack/seaweedfs-naming-audit.sh, and the copy that shipped in v1.6.0 silences
every kubectl call with 2>/dev/null. A timeout or an RBAC denial therefore
prints an empty "all clean" table indistinguishable from a genuinely clean
fleet, and that output gates a runbook step that deletes PVCs. Anyone
following those notes today can still get a false clean.

Verified per tag: v1.6.0 has 11 blanket redirections and no run_kubectl
wrapper; v1.6.1, v1.6.2 and v1.5.4 have 4 and 18 respectively, the
fail-closed shape from #3436 and its backport #3474.

Published changelogs are historical records and are not normally edited.
This one is an exception on purpose, because the instruction it carries is
still live and still wrong.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
myasnikovdaniil added a commit that referenced this pull request Aug 19, 2026
…udit note (#3910)

Adds the changelog for `v1.5.4`, and corrects one still-live instruction
in v1.6.0's.

The generated changelog is a good inventory and every one of its bullets
is kept. What it had no layer for is the operator: no required-actions
section, no runnable pre-upgrade checks, and the SeaweedFS 4.31 rename
left as three isolated fix bullets that never say what to do about it.
That is what is added on top, in the shape v1.6.0's changelog uses.

Base widened to `v1.5.2`. v1.5.3 was tagged but its release was never
published and its changelog PR was closed unmerged, so
`docs/changelogs/v1.5.3.md` exists on no ref and a v1.5.3 base leaves
#3212 (persistent EFI/TPM state) and #3194 (filer postgres2 connection
pool) documented nowhere at all. Coverage is 82/82 commits in range.

Three claims corrected. The slot 45 divergence is a skip rather than an
ordering problem: v1.5.4 is the first 1.5.x release stamped
`targetVersion: 46`, and `run-migrations.sh` loops `seq CURRENT (TARGET
- 1)`, so a cluster that reaches 46 runs slots 46 through 53 on the way
to 1.6 and never executes 1.6's own slot 45. Its chart-side half was
missing too and is not redundant, because a fresh v1.5.4 install is
stamped 46 having never run any slot, so only the chart can reach that
population. And the S3 checksum bullet named a barman-cloud plugin path
that does not exist on `release-1.5`.

The audit re-run warning names v1.6.0 rather than an earlier release,
because the script does not exist at v1.5.2 at all. v1.5.4 is the first
1.5.x release to carry it, and it carries the fail-closed version.

The second commit edits `docs/changelogs/v1.6.0.md`, which is deliberate
rather than an accident of scope. Published changelogs are historical
records and are normally left alone, but v1.6.0's upgrade notes still
tell operators to run `hack/seaweedfs-naming-audit.sh`, and the copy
that shipped in v1.6.0 silences every `kubectl` call with `2>/dev/null`.
A timeout or an RBAC denial therefore prints an empty all-clean table
indistinguishable from a genuinely clean fleet, and that output gates a
runbook step that deletes PVCs. Verified per tag: v1.6.0 carries 11
blanket redirections and no `run_kubectl` wrapper, while v1.6.1, v1.6.2
and v1.5.4 carry 4 and 18, the fail-closed shape from #3436 and its
backport #3474.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) area/testing Issues or PRs related to testing (e2e, bats, unit tests) backport Should change be backported on previous release kind/bug Categorizes issue or PR as related to a bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant