Skip to content

fix(kubernetes): Delete orphaned resources in new post-delete cleanup - #3855

Closed
Ray Low (crossRT) wants to merge 30 commits into
cozystack:mainfrom
crossRT:delete-k8s-orphaned-resources
Closed

fix(kubernetes): Delete orphaned resources in new post-delete cleanup#3855
Ray Low (crossRT) wants to merge 30 commits into
cozystack:mainfrom
crossRT:delete-k8s-orphaned-resources

Conversation

@crossRT

@crossRT Ray Low (crossRT) commented Aug 16, 2026

Copy link
Copy Markdown

What this PR does

When tenant deletes the kubernetes cluster, 2 resources are orphaned and leftover in the tenant namespace.

  • jobs/-oidc-bootstrap
  • secrets/-talos-ca
  • secrets/-talos-tls-cert

This PR adds a new post-delete cleanup job and delete these resources manually.

Downstream repositories

Release note

fix(kubernetes): Fixed the `<name>-talos-ca` and `<name>-talos-tls-cert` Secrets and the `<name>-oidc-bootstrap` Job being left in the tenant namespace after a Kubernetes cluster is deleted.

Summary by CodeRabbit

  • New Features

    • Added automated builds and publishing for maintained release branches.
    • Added post-delete cleanup for orphaned Kubernetes resources.
    • Added CloudNativePG extension configuration support.
  • Bug Fixes

    • Improved branch-specific artifact promotion and release changelog handling.
    • Prevented unsafe image overwrites during promotion.
    • Improved SeaweedFS audit failure detection and recovery guidance.
    • Ensured default Kubernetes node groups reconcile correctly.
    • Added compatibility checks for hostname policies.
  • Updates

    • Updated platform components and container images to v1.6.1.
    • Updated etcd-operator and CloudNativePG versions.
  • Documentation

    • Added the v1.6.1 changelog.

Render the route, gateway and ingress hostname ValidatingAdmissionPolicies
only where the admissionregistration.k8s.io/v1/ValidatingAdmissionPolicy API
is served, matching the deletion-protection guard, so a cluster without that
API does not receive an unrenderable resource.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 267ea51)
skopeo inspect --format '{{.Digest}}' fails on OCI artifacts: for
cozystack-packages (config.mediaType
application/vnd.cncf.flux.config.v1+json) it prints nothing and exits
nonzero, so the v1.6.0 finalize aborted its post-copy verification right
after both copies had already succeeded, leaving 32 of 43 repos without
stable tags and skipping the cozy-installer publish. --dry-run never
executes this branch, so rehearsals could not catch it.

Compute the digest as sha256 of the raw manifest instead, which is how
registries define it and works for any media type. The helper gates the
hash on skopeo's exit status via a temp file rather than a pipeline:
a missing tag still yields an empty string (pre-check reads that as
"tag absent, proceed to copy"), never the empty-input hash
sha256:e3b0c442..., and never an abort under set -eu.

Verified against live GHCR: the raw-manifest hash of
cozystack-packages:v1.6.0 matches the digest finalize expected
(bf68208730860fa8...), and agrees byte-for-byte with the old method on
regular multi-arch images. New cozytest cases cover the OCI-artifact
digest, a post-copy mismatch, and the missing-tag path.

Assisted-By: GPT-5 <noreply@openai.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5d8dd51)
Review notes from cozystack#3435, and a correction to my first attempt at them.

manifest_digest gated on skopeo`s exit status alone, so a zero exit with no
bytes hashed the empty string to sha256:e3b0c442... — a digest that looks real
and belongs to no manifest. The first fix simply required non-empty bytes, which
was worse than it looked: the caller turns an empty result into "not published,
safe to copy", so a registry answering 200 with an empty body went from an
accidental refusal (the bogus digest mismatched, so the write-once check
refused) to an actual copy over whatever the tag really held. Verified against
real skopeo 1.23: a 200 with Content-Length 0 does exit 0 with no stdout.

The helper is now explicitly three-state. A digest, or empty output for a tag the
registry PROVED absent, or non-zero for anything indeterminate — which under
set -e aborts the promotion before it writes. Absence has to be reported as
"manifest unknown"; 429, 5xx, auth and network failures all exit non-zero with
no bytes too, and reading those as "unpublished" is the same fail-open one level
down. finalize retags ~42 refs with no retry wrapper, so a single rate-limit was
enough to reach that path. skopeo`s stderr is captured instead of discarded, so
the diagnostics now name the real failure.

Tests: the write-once refusal branch the review found untested (an existing
stable tag at a different digest must be refused before any write), the empty-
body case (refuses, writes nothing, and the empty-input hash never appears in
the diagnostics), and a 429 that must not read as an absent tag. The mock now
emits realistic registry errors, since a bare non-zero exit is no longer proof
of absence. Both new guards are mutation-checked: accepting any non-zero exit as
absence fails the 429 test, dropping the empty-bytes check fails the empty-body
test.

11/11 tests green, hack/image-pin-consistency.bats still 2/2, sh -n clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit ae39475)
…errors

hack/seaweedfs-naming-audit.sh was fail-open: any kubectl failure
produced an empty table indistinguishable from an honestly clean fleet
(namespace LIST failure = zero namespaces walked, secret/pvc/sts LIST
failures = zero findings, all under 2>/dev/null with no error handling).
The runbook uses this output as the gate before PVC deletion, as
post-deletion verification, and as an upgrade precondition, so a
transient API error could green-light destroying data (cozystack#3431).

Route every kubectl call through a run_kubectl helper: non-zero exit
prints a FATAL line naming the failed query and propagates the code up
the whole chain (enumerations restructured to capture-then-check, since
an exit inside $(...) dies with the subshell). By-name GETs distinguish
legitimate absence from real errors via --ignore-not-found on a
separate existence check; a Secret that exists but has no decodable
release payload, or one that decodes without a chart name, is corrupt
state and fails loudly instead of silently dropping the tenant.
Incomplete PV-age evidence now marks the generation incomplete and
falls to the safe no-direction branch, so a failed GET can no longer
flip OVERLAP into a wrong Step-3 deletion candidate. A successful run
with zero findings still prints the same bytes as before.

Tests: 14 new cozytest cases — failure injection for every LIST and
by-name GET site, corrupt-payload variants, and full-output golden
diffs for the success paths (verified byte-identical to the pre-change
script). 25/25 green.

Fixes cozystack#3431

Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 3dc6cbd)
…odelled calls

Review findings from cozystack#3436, plus what an adversarial re-check of the first
attempt at them turned up.

The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals
"config" (the release`s values) after "chart", so a values subtree that spells
chart.metadata.name shadowed the real chart name: the release then read as
non-SeaweedFS, the tenant silently vanished from the report, and the script
exited 0 -- the precise false clean this PR exists to prevent, reachable
without any corruption at all. It now takes the FIRST match. The path stays
adjacent on purpose: a looser "any name after metadata" matches
chart.templates[].name, which Helm emits immediately after metadata on every
healthy release, so the review`s suggested relaxation would have returned a
template path for every tenant. Over-strictness fails loudly and recoverably;
over-looseness deletes data quietly.

Newlines are now folded before matching, so a pretty-printed payload parses at
all rather than aborting, and whitespace around the punctuation is tolerated.
first_deployed got the same treatment -- it had the identical greedy, single-
line-only shape, and a spaced payload silently dropped the PV-vintage row from
the report.

The FATAL message names the path it read and the shape it expected, so a future
Helm format change is diagnosable instead of looking like real corruption.

Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy)
each byte-compared against the same golden as the compact payload, so the shape
must make no difference to the report. Mutation-checked -- restoring last-match
extraction fails the decoy test on its own, restoring the single-line matcher
fails the pretty test.

The test fake answered any unmodelled kubectl invocation with exit 0 and empty
stdout, the fail-open shape this script was rewritten to reject, and enough to
let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97
naming the invocation. Verified inert first by instrumenting the fake: 88
invocations across the suite, none unmodelled. Also added the missing test for
the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it.

The runbook now tells the operator to read the exit code: a non-zero exit means
the table is incomplete and no step may be taken on it. That contract lived only
in the script header, while the runbook is what the operator follows.

29/29 tests green; sh -n clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit e02adad)
…on kubectl and payload errors (cozystack#3474)

# Description
Backport of cozystack#3436 to `release-1.6`.
…cation media-type-agnostic (cozystack#3473)

# Description
Backport of cozystack#3435 to `release-1.6`.
…ardown

The pre-delete teardown Job clears the HelmRelease finalizers as its
final step, but targeted a hardcoded namespace that does not match the
namespace where the release and its RBAC live. The Job's Role and
RoleBinding are created in the release namespace and grant patch on the
HelmRelease by release name, so the ServiceAccount was Forbidden to patch
the HelmRelease in the hardcoded namespace. The Job errored and retried
forever, the HelmRelease stuck in Terminating, and the Helm release
wedged in "uninstalling", blocking any teardown or reinstall.

Target the release name and namespace so the patch matches the RBAC that
grants it. Add a helm-unittest pinning the rendered namespace.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit fd568de)
…olicies on the VAP API (cozystack#3442)

# Description
Backport of cozystack#3409 to `release-1.6`.
The PR finalize job pulls cozystack-packages:main and repoints every package
the PR did not rebuild at current-main images. On a main-based PR that is the
whole point: it stops e2e from testing last-release images for everything
outside the build matrix. On a release-line PR it is wrong — the committed refs
there are not stale, they are that line`s released digests, and the charts are
written against exactly those.

So a release-line PR was installing main`s binaries onto its own charts, and the
mismatch grows with every commit main gains. cozystack#3437 is the demonstration: a
one-line change on release-1.6 that deactivates an app failed install
deterministically, twice, with

  SchemaError(...core/v1alpha1.Option.spec): unknown model in reference:
  "...core~1v1alpha1.OptionSpec"

from main`s cozystack-controller serving an aggregated OpenAPI that branch`s
charts cannot validate against. Nothing in the PR was broken; the lane was.
Left alone this makes every 1.6 backport look red, which is when release-branch
PRs are busiest.

Both overlay steps are now gated on `github.base_ref == main`, so a release-line
PR keeps its committed digests — the behaviour that predates the overlay.
Retargeting the overlay at a per-line artifact would be better but is not
possible today: build-main.yaml publishes only cozystack-packages:main, and the
registry carries no release-* equivalent (verified against the packages repo`s
tag list: `main` plus per-PR tags, nothing else).

hack/overlay-main-images_test.bats pins the wiring per step, so adding a third
overlay step without the guard fails the suite. Mutation-checked by removing one
guard. 12/12 green; actionlint clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit fea40d6)
… artifacts

Supersedes the interim guard in this branch`s first commit, which skipped the
overlay entirely for release-line PRs. Skipping fixed the wrong images but left
those PRs testing their line`s last release: for any package the PR did not
rebuild, the committed ref is the released digest, so a component changed by an
earlier backport was exercised as its pre-backport binary until the next rc.

The overlay now reads `cozystack-packages:<base branch>` instead of always
`:main`, and build-release.yaml publishes that artifact for every maintained
`release-<major>.<minor>` branch the way build-main.yaml does for main: images
tagged with the branch, and the whole packages tree pushed with each reference
digest-pinned to what the run just built. Each base branch therefore has its own
generation to overlay from, which is what the original bug was really about —
cozystack#3437 failed install because main`s cozystack-controller served an aggregated
OpenAPI release-1.6`s charts could not validate against.

Three deliberate choices:

  * The trigger matches line branches only (`release-[0-9]+.[0-9]+`). The
    per-release and rc staging branches promote-rc.yaml and tags.yaml create
    (release-1.6.1, release-1.6.0-rc.4) must not trigger a full rebuild — their
    images come from the tag build, and rebuilding them would be waste.
  * WRITE_CACHE stays 0. CACHE_REGISTRY/<img>:buildcache is a single ref per
    image and build-main.yaml is deliberately its only, serialized writer so
    concurrent builds cannot race on the cache manifest (the 409 class cozystack#2711
    fixed for image tags). A line build can overlap a main build, so writing here
    would reintroduce that race. Line builds read the cache.
  * A missing artifact still degrades to committed refs, but on a release line it
    now emits a ::warning:: naming the branch. Silent degradation is
    indistinguishable from a working overlay, which is how a mis-specified branch
    filter would hide for a whole release cycle.

Cost: one `make build` per push to a maintained line, i.e. per merged backport.

hack/overlay-main-images_test.bats pins the artifact tag to the base branch,
rejects a hardcoded :main in either overlay step, and pins build-release.yaml`s
branch filter, image tag and WRITE_CACHE=0. Mutation-checked: restoring :main,
setting WRITE_CACHE=1, and broadening the filter to release-* each fail a test.
13/13 green; actionlint and zizmor clean.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 2330d6f)
…h, and publish per-line artifacts (cozystack#3514)

## What this PR does

Hand backport of cozystack#3471 to `release-1.6`. Two clean cherry-picks, each
carrying its `-x` reference; the resulting tree is byte-identical to
cozystack#3471's merged state for all three files.

The overlay must be fixed **on this branch** to have any effect here.
For `pull_request` events GitHub builds the workflow from the merge ref,
and for `push` events it reads the workflow from the pushed ref — so
neither half of cozystack#3471 reaches this line while it lives only on `main`.
Concretely, `build-release.yaml` on `main` never fires for a push to
`release-1.6`, and a `release-1.6` PR keeps using this branch's copy of
`pull-requests.yaml`.

## Why the bot could not do this

The automatic backport reported success on every job and opened nothing
(run 30817682877). It is not a conflict — cozystack#3471 carries an empty commit,
`007d0b1a` (`chore(ci): re-trigger CI after a label event produced a
no-op run`, zero files). `korthout/backport-action` cherry-picks
commit-by-commit without `--allow-empty`, so that commit fails with `The
previous cherry-pick is now empty`; the `draft_commit_conflicts`
fallback then runs `git commit --all -m BACKPORT-CONFLICT`, which also
has nothing to commit, and the action aborts:

```
git cherry-pick -x 007d0b1
The previous cherry-pick is now empty, possibly due to conflict resolution.
git commit --all -m BACKPORT-CONFLICT
The previous cherry-pick is now empty, possibly due to conflict resolution.
git cherry-pick --abort
Backport failed for `release-1.6`, because it was unable to cherry-pick the commit(s).
```

This is a general gap, not specific to this PR: any labelled PR
containing an empty commit backports to nothing, and the failure
surfaces as three green jobs. Worth a follow-up on the workflow —
skipping empty commits, or having the action's failure fail the job —
but that is out of scope here.

## What this unblocks

cozystack#3437 fails install deterministically on this line today, because the
unguarded overlay hands it main's `cozystack-controller` to run against
`release-1.6`'s charts. The binary indexes
`internal.cozystack.io/v1alpha1 TenantProjection` at startup, this
branch ships no such CRD, so it fatals and every release that depends on
it stalls. With this merged, the overlay reads
`cozystack-packages:release-1.6` instead.

That works even before the first `build-release.yaml` run publishes the
artifact: a missing artifact degrades to the committed refs, which on
this branch are the v1.6.0 digests its charts are written against. The
degraded path is also no longer silent — it emits a warning naming the
branch.

cozystack#3437 will need a new head commit rather than a re-run, since the
finalize job is reused on re-run and its existing `pr.patch` already has
main's refs baked in.

## Note on cost

Merging this is itself a push to `release-1.6`, so `build-release.yaml`
fires on it and runs a full `make build` (up to 2h on a 24-cpu runner)
to publish the line's images and `cozystack-packages:release-1.6`. That
recurs per push to this line; `concurrency` with `cancel-in-progress`
collapses bursts, and `paths-ignore` only exempts `docs/**`.

## Verification

`hack/overlay-main-images_test.bats` 13/13 green on this branch,
including the two wiring tests the backport brings (`the overlay reads
the artifact for the PR's own base branch`, `every maintained release
line publishes its own packages artifact`). `actionlint` clean on both
workflows, and both parse as YAML. `hack/common-envs.mk` is
byte-identical between `main` and `release-1.6`, so `IMAGE_TAG`,
`WRITE_CACHE`, `PUBLISH_VERSIONED` and `PUBLISH_FLOATING` behave here
exactly as they do on main.

### Release note

```release-note
NONE
```
kubernetes-nodes isn't ready to launch on 1.6 yet. Remove only its include
from the iaas platform bundle so the package still ships but stays inactive;
restore the line when the feature is ready. The package, API types and RD
package are kept intact.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…ne (cozystack#3437)

## What this changes

`kubernetes-nodes` is not ready to launch on the 1.6 line yet, so this
removes its single include from the `iaas` platform bundle
(`packages/core/platform/templates/bundles/iaas.yaml`). The platform
stops activating the package while it still ships in the release; the
line comes back when the feature is ready.

## Scope

One line removed, nothing else. The `kubernetes-nodes` app package, its
API types (`api/apps/v1alpha1/kubernetesnodes`), and the
`kubernetes-nodes-rd` system package are all kept intact — this
deactivates the feature, it does not delete it.

```release-note
NONE
```
…release namespace on teardown (cozystack#3478)

# Description
Backport of cozystack#3372 to `release-1.6`.
release-1.6 was cut at v1.6.0 on 2026-07-22, and the finalize fixes
landed on main afterwards, so shipping v1.6.1 from this line would repeat
two defects v1.6.0 hit. Workflows run from the ref they fire on: a promote
PR based on release-1.6 runs THIS file, not main's.

Carries three upstream commits, after which this file is byte-identical
to main, so future backports touching it will not conflict:

  01e1e71  persist-credentials: false on the checkout
  f49d54a  publish the release with the merged changelog as its body
  ba67fea  drop paths-ignore from the trigger

Without the first, the checkout persists GITHUB_TOKEN as an
http.extraheader that silently wins over the app token injected by
git remote set-url. The stable tag then pushes as GITHUB_TOKEN, which
creates no workflow run, so tags.yaml never fires and its
generate-changelog and update-website-docs backstops stay silent. That is
exactly what happened to v1.6.0.

Without the second, the release publishes with the draft's body
("Promoted from vX.Y.Z-rc.N"). The two compound on a maintenance line: a
patch's changelog is committed to release-1.6 and never reaches main, and
update-releasenotes.yaml only watches main, so nothing would ever sync it.
v1.6.1 would ship with placeholder release notes permanently.

The third is latent rather than active — a promote PR carrying only
docs/changelogs/vX.Y.Z.md would be dropped by the filter, producing no
finalize run, no tag and no error. v1.6.1 will carry tag-string rewrites
so it would not have fired, but the filter has no remaining purpose now
that the promote PR always carries a changelog.

actionlint and zizmor clean; no bats suite references this workflow, and
hack/promote-gate-contract.bats does not exist on this line.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## Why

`release-1.6` was created at v1.6.0's merge commit on 2026-07-22, and
the finalize fixes landed on `main` after that. Workflow files run from
the ref they fire on, so a promote PR based on `release-1.6` runs
**this** copy of `pull-requests-release.yaml`, not `main`'s. Shipping
v1.6.1 from this line today would repeat two defects v1.6.0 already hit.

## What this carries

Three upstream commits, after which this file is **byte-identical to
`main`'s copy** — so later backports touching it will not conflict.

| Upstream | Change |
| --- | --- |
| `01e1e7188` | `persist-credentials: false` on the checkout |
| `f49d54a68` | publish the release with the merged changelog as its
body |
| `ba67fea7d` | drop `paths-ignore` from the trigger |

### persist-credentials

Without it the checkout persists `GITHUB_TOKEN` as an
`http.extraheader`, which silently wins over the app token injected by
`git remote set-url`. The stable tag then pushes as `GITHUB_TOKEN`,
which creates no workflow run, so `tags.yaml` never fires and its
`generate-changelog` and `update-website-docs` backstops stay silent.
That is exactly what happened to v1.6.0.

### Changelog as release body

Without it the release publishes with the draft's own body, `"Promoted
from vX.Y.Z-rc.N"`.

**These two compound into a permanent defect on a maintenance line.** A
patch's changelog is committed to `release-1.6.1`, merged into
`release-1.6`, and never reaches `main` — and `update-releasenotes.yaml`
only watches `main`, so nothing would ever sync it. The tag-time
backstop that would have ported it is dead because of the credential
bug. v1.6.1 would ship with placeholder release notes permanently, with
no error anywhere.

### paths-ignore

Latent rather than active. A promote PR carrying only
`docs/changelogs/vX.Y.Z.md` would be dropped by the filter, producing no
finalize run, no tag and no error. v1.6.1 will carry tag-string rewrites
so it would not have fired, but the filter has no remaining purpose now
that the promote PR always carries a changelog.

## On the cherry-pick policy

`docs/release.md`'s skip rule says CI-only changes do not belong in a
patch. That rule governs release *contents*; backporting release
*machinery* so the line is releasable at all is a different thing, and
there is precedent already on this branch — cozystack#3473, cozystack#3474 and cozystack#3514 are
all CI/release backports.

## Verification

`actionlint` clean. `zizmor` clean, no findings. YAML parses. No bats
suite references this workflow, and `hack/promote-gate-contract.bats`
does not exist on this line, so there is no stale contract test to trip.
The diff against `origin/main`'s copy of the file is empty.

## Not included

Three known gaps on this line are deliberately out of scope, none of
which needs a code change to ship v1.6.1:

- `e2e-tag.yaml` is absent and `tags.yaml` has no `rc-e2e` job, so no
E2E runs on the rc. Handled by dispatching **E2E Release Tag from
`main`** against the published rc tag, which also supplies the promote
gate's alternate evidence.
- `pull-requests.yaml` has no `labeled` trigger, so adding `full-e2e` to
the promote PR starts nothing. Backporting that needs the label-event
guards too, and rc-time validation makes it unnecessary here.
- finalize is still the pre-cozystack#3456 monolith, so a mid-registry failure is
not re-runnable and needs the documented hand recovery.

```release-note
NONE
```
…roup

The Job in templates/talos/talos-reconcile-job.yaml is the only
producer of the TalosConfigTemplate that each worker
MachineDeployment names in spec.template.spec.bootstrap.configRef;
cluster.yaml renders the reference but deliberately not the object.
Its loop read .Values.nodeGroups directly, while the
MachineDeployment loop reads the effective set through the
kubernetes.nodeGroups helper.

For a cluster that supplies no nodeGroups the two disagree: the
helper's else-branch emits the built-in md0 group, so the md0
MachineDeployment renders, but the raw map is empty and no Job
renders at all. Every Machine the cluster-autoscaler adds to md0
then blocks indefinitely on a TalosConfigTemplate that nothing will
ever create, and the KamajiControlPlane spec.network.certSANs patch
the same Job performs is skipped too.

Range over the helper so the Job set tracks the MachineDeployment
set exactly. No change for a cluster that declares its own groups,
and md0 stays removable: the helper's if-branch keeps a
user-supplied map authoritative.

The gap survived because every helm-unittest fixture and e2e suite
declares md0 explicitly. tests/nodegroups_default_test.yaml does
render the empty-nodeGroups case but lists only
templates/cluster.yaml, so it never looked at the Job. The new
suite pins both halves of the helper contract and fails without
this fix.

Fixes cozystack#3504

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 4e9c5ce)
…b for the default md0 group (cozystack#3536)

# Description
Backport of cozystack#3535 to `release-1.6`.
Bump the cozystack etcd-operator packages from v0.5.3 to v0.5.4. The
release is a controller bug-fix set with no API, RBAC or values changes:

- fix(controllers): derive --initial-cluster-state from phase, not from
  the seed (cozystack/etcd-operator#355)
- fix(controllers): stop exempting the bootstrap seed from self-heal
  (cozystack/etcd-operator#354)
- fix(controllers): extend crash-loop self-heal to memory members
  (cozystack/etcd-operator#352)
- fix(controllers): switch the PDB from maxUnavailable to minAvailable
  (cozystack/etcd-operator#351)

Adaptations:
- etcd-operator/Chart.yaml: appVersion v0.5.3 -> v0.5.4 (the manager image
  tag defaults to .Chart.AppVersion, so this reimages the controller).
- etcd-operator/Makefile, etcd-operator-crds/Makefile: ETCD_OPERATOR_REF
  v0.5.3 -> v0.5.4.
- etcd-operator-crds/templates/etcdmembers.yaml: re-vendored at v0.5.4 via
  `make update`; description-only change to the /scale replicas/selector
  fields tracking the PDB minAvailable fix. etcdclusters/etcdsnapshots
  unchanged.
- templates/rbac.yaml left as-is: manager-role-rules.yaml is byte-identical
  between v0.5.3 and v0.5.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
(cherry picked from commit 19f969b)
The manager image tag derives from .Chart.AppVersion, so bumping appVersion
to v0.5.4 without updating the two image literals in the deployment unittest
broke the suite (helm-unittest gates the PR via `make unit-tests`). Update
both asserts (manager container image and the agent OPERATOR_IMAGE env) to
ghcr.io/cozystack/etcd-operator:v0.5.4. Suite is green again (18/18).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
(cherry picked from commit 22c1df8)
…C resize-deadlock fix

Raises the CloudNativePG operator image and its CRDs together to 1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 / #9981): after a simultaneous resources+size change the operator deletes the sole primary Pod, leaves the PVC in the resizing class, and never recreates the Pod, wedging the cluster.

Upstream publishes a chart only per minor .0/.1, so there is no 1.28.2 chart on the 1.28 line. Following cozystack#3526/cozystack#3528, patches/cloudnative-pg-1.28.2.patch (applied by make update) raises the vendored chart's appVersion and CRDs from 1.28.1 to 1.28.2 in lockstep; the operator image follows appVersion, so no image.tag pin. Verified cozystack#3479-safe: no CRD status field changes between 1.28.1 and 1.28.2 (only the extensions spec grows). Backportable to release-1.6.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit bcb3626)
…d CRDs to 1.28.2 for PVC resize-deadlock fix (cozystack#3542)

# Description
Backport of cozystack#3510 to `release-1.6`.
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Promotes `v1.6.1-rc.1` to stable `v1.6.1` — no rebuild. The tree pins
the rc's images by digest. On merge, pull-requests-release.yaml creates
the write-once `v1.6.1` tag at the merge commit, retags those digests to
`v1.6.1` (+`:latest` when this is the newest stable), publishes the
stable cozy-installer chart, and publishes the release — so the stable
artifacts are bit-for-bit the rc that passed e2e. Do NOT squash-merge
(decision B): the stable tag must attach to a real merge commit.

⚠️ **RC e2e gate bypassed** — this promotion did not verify green full
e2e evidence for `v1.6.1-rc.1`.

ℹ️ E2E already ran against the immutable rc, so this promote PR does not
run E2E by default. A maintainer may add the `full-e2e` label to run the
full suite again.

✅ Includes `docs/changelogs/v1.6.1.md`, which finalize uses verbatim as
the published release body.

✅ Website docs PR opened/refreshed on `cozystack/website` (branch
`update-docs-v1.6.1`), generated from the `release-1.6.1` staging
branch. **Do NOT merge that PR until `v1.6.1` is published** — merging
it early flips the site's latest-version pointer to an unpublished
version and 404s its docs/API links (see the PR body).
… and oidc-bootstrap job) in new post-delete cleanup
@github-actions github-actions Bot added area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/XXL This PR changes 1000+ lines, ignoring generated files labels Aug 16, 2026
@coderabbitai

coderabbitai Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 4cd679e7-87a4-4a33-871c-b924601fe9e1

📥 Commits

Reviewing files that changed from the base of the PR and between 9e3f66f and 8ca08b6.

⛔ Files ignored due to path filters (1)
  • packages/system/capi-providers-cpprovider/files/components.gz is excluded by !**/*.gz
📒 Files selected for processing (72)
  • .github/workflows/build-release.yaml
  • .github/workflows/pull-requests-release.yaml
  • .github/workflows/pull-requests.yaml
  • docs/changelogs/v1.6.1.md
  • docs/operations/seaweedfs-431-rename-recovery.md
  • hack/overlay-main-images_test.bats
  • hack/promote-retag.sh
  • hack/promote-retag_test.bats
  • hack/seaweedfs-naming-audit.bats
  • hack/seaweedfs-naming-audit.sh
  • packages/apps/clickhouse/images/altinity-clickhouse-backup.tag
  • packages/apps/clickhouse/images/clickhouse-backup.tag
  • packages/apps/http-cache/images/nginx-cache.tag
  • packages/apps/kubernetes/images/cluster-autoscaler.tag
  • packages/apps/kubernetes/images/kubevirt-cloud-provider.tag
  • packages/apps/kubernetes/images/kubevirt-csi-driver.tag
  • packages/apps/kubernetes/images/talos-csr-signer.tag
  • packages/apps/kubernetes/templates/post-delete.yaml
  • packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml
  • packages/apps/kubernetes/tests/post_delete_cleanup_test.yaml
  • packages/apps/kubernetes/tests/talos_reconcile_nodegroups_test.yaml
  • packages/apps/mariadb/images/mariadb-backup.tag
  • packages/core/installer/values.yaml
  • packages/core/platform/templates/bundles/iaas.yaml
  • packages/core/platform/values.yaml
  • packages/core/testing/values.yaml
  • packages/extra/bootbox/images/matchbox.tag
  • packages/extra/seaweedfs/images/objectstorage-sidecar.tag
  • packages/system/backup-controller/values.yaml
  • packages/system/backupstrategy-controller/values.yaml
  • packages/system/bucket/images/s3manager.tag
  • packages/system/capi-providers-cpprovider/files/control-plane-components.yaml
  • packages/system/capi-providers-cpprovider/images/cluster-api-control-plane-provider-kamaji.tag
  • packages/system/cilium/values.yaml
  • packages/system/cozystack-api/values.yaml
  • packages/system/cozystack-basics/templates/gateway-hostname-policy.yaml
  • packages/system/cozystack-basics/templates/ingress-hostname-policy.yaml
  • packages/system/cozystack-basics/templates/route-hostname-policy.yaml
  • packages/system/cozystack-basics/tests/gateway-hostname-policy_test.yaml
  • packages/system/cozystack-basics/tests/hostname-policies-capability-gate_test.yaml
  • packages/system/cozystack-basics/tests/ingress-hostname-policy_test.yaml
  • packages/system/cozystack-basics/tests/route-hostname-policy_test.yaml
  • packages/system/cozystack-controller/values.yaml
  • packages/system/dashboard/values.yaml
  • packages/system/etcd-operator-crds/Makefile
  • packages/system/etcd-operator-crds/templates/etcdmembers.yaml
  • packages/system/etcd-operator/Chart.yaml
  • packages/system/etcd-operator/Makefile
  • packages/system/etcd-operator/tests/deployment_test.yaml
  • packages/system/flux-shard-operator/values.yaml
  • packages/system/grafana-operator/images/grafana-dashboards.tag
  • packages/system/kamaji/values.yaml
  • packages/system/keycloak-configure/templates/delete.yaml
  • packages/system/keycloak-configure/tests/delete_test.yaml
  • packages/system/kubeovn-plunger/values.yaml
  • packages/system/kubeovn-webhook/values.yaml
  • packages/system/kubevirt-csi-node/values.yaml
  • packages/system/lineage-controller-webhook/values.yaml
  • packages/system/linstor-gui/values.yaml
  • packages/system/linstor/values.yaml
  • packages/system/metallb/values.yaml
  • packages/system/monitoring/images/grafana.tag
  • packages/system/multus/templates/multus-daemonset-thick.yml
  • packages/system/objectstorage-controller/values.yaml
  • packages/system/postgres-operator/Makefile
  • packages/system/postgres-operator/charts/cloudnative-pg/Chart.yaml
  • packages/system/postgres-operator/charts/cloudnative-pg/templates/crds/crds.yaml
  • packages/system/postgres-operator/patches/cloudnative-pg-1.28.2.patch
  • packages/system/postgres-operator/tests/cnpg-version_test.yaml
  • packages/system/postgres-operator/values.yaml
  • packages/system/seaweedfs/values.yaml
  • packages/system/securitygroup-controller/values.yaml

📝 Walkthrough

Walkthrough

The PR prepares the v1.6.1 release. It adds release-line artifact publishing, hardens registry promotion and SeaweedFS audits, updates Kubernetes chart behavior, refreshes operator and image versions, and adds release documentation.

Changes

Release and promotion

Layer / File(s) Summary
Release-line workflows and branch artifacts
.github/workflows/build-release.yaml, .github/workflows/pull-requests-release.yaml, .github/workflows/pull-requests.yaml, hack/overlay-main-images_test.bats, docs/changelogs/v1.6.1.md
Release builds publish branch-named images and package artifacts. Release finalization loads the merge-commit changelog. PR overlays use the base branch artifact.
Raw manifest promotion checks
hack/promote-retag.sh, hack/promote-retag_test.bats
Promotion uses raw manifest digests and fails on ambiguous registry responses. Tests cover missing, conflicting, empty, transient, and post-copy verification cases.

Audit hardening

Layer / File(s) Summary
Fail-closed SeaweedFS audit
hack/seaweedfs-naming-audit.sh, hack/seaweedfs-naming-audit.bats, docs/operations/seaweedfs-431-rename-recovery.md
The audit reports kubectl and payload errors, distinguishes absent objects from failures, validates PV evidence, and stops on incomplete state.

Kubernetes package behavior

Layer / File(s) Summary
Cleanup and node-group jobs
packages/apps/kubernetes/templates/post-delete.yaml, packages/apps/kubernetes/templates/talos/talos-reconcile-job.yaml, packages/apps/kubernetes/tests/*cleanup*, packages/apps/kubernetes/tests/*nodegroups*
A restricted post-delete Job removes named orphaned resources. Talos reconciliation uses effective node groups, including the default md0 group.
Capability gates and release-aware teardown
packages/system/cozystack-basics/templates/*hostname-policy.yaml, packages/system/cozystack-basics/tests/*, packages/system/keycloak-configure/templates/delete.yaml, packages/system/keycloak-configure/tests/delete_test.yaml
Hostname policies require ValidatingAdmissionPolicy support. Keycloak teardown uses the current release name and namespace.

Dependency and package refresh

Layer / File(s) Summary
Operator versions and CRD schemas
packages/system/etcd-operator-*, packages/system/postgres-operator/*
The etcd operator moves to v0.5.4. CloudNativePG moves to 1.28.2 with patched CRD schemas and alignment tests.
Image pins and package wiring
packages/apps/*/images/*, packages/core/*, packages/extra/*, packages/system/*
Package images and selected platform references move to v1.6.1 or new immutable digests. The IaaS bundle removes the Kubernetes nodes application include.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related issues

Possibly related PRs

Suggested labels: kind/cleanup, area/kubernetes

Suggested reviewers: ivanhunters, lexfrei

✨ Finishing Touches 💡 1
⚔️ Resolve merge conflicts 💡
  • Resolve merge conflict in branch delete-k8s-orphaned-resources
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Warning

⚠️ This pull request has been flagged as potential spam (other-spam) by CodeRabbit slop detection and should be reviewed carefully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kubernetes Issues or PRs related to the tenant Kubernetes app kind/bug Categorizes issue or PR as related to a bug size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants