fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090) - #3170
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a resource leakage issue where etcd data PVCs and TLS secrets persist in the tenant namespace after the etcd module is disabled. By implementing a hardened post-delete cleanup Job, the system now ensures these orphaned resources are garbage-collected, preventing storage and secret accumulation. Note that this change results in the permanent deletion of etcd data upon disabling the module. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThis PR adds a post-delete etcd cleanup hook that deletes orphaned PVCs and TLS secrets, with scoped RBAC, a hardened Job, helm-unittest coverage, and a phony Makefile test target. ChangesCleanup hook implementation and tests
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Helm
participant ServiceAccount
participant Role
participant RoleBinding
participant CleanupJob
participant KubeAPI
Helm->>ServiceAccount: render post-delete hook identity
Helm->>Role: render scoped delete permissions
Helm->>RoleBinding: bind Role to ServiceAccount
Helm->>CleanupJob: trigger post-delete hook
CleanupJob->>KubeAPI: delete data-etcd-<n> PVCs
CleanupJob->>KubeAPI: delete named TLS secrets
KubeAPI-->>CleanupJob: deletion result
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Warning Gemini encountered an error creating the review. You can try again by commenting |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/extra/etcd/tests/cleanup_test.yaml (1)
46-48: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert the Role and RoleBinding are actual post-delete hooks.
These tests only check hook weight. Add assertions for
metadata.annotations["helm.sh/hook"]andmetadata.annotations["helm.sh/hook-delete-policy"]so a regression cannot render the RBAC as normal release resources.Suggested test coverage addition
- equal: + path: metadata.annotations["helm.sh/hook"] + value: post-delete + - equal: path: metadata.annotations["helm.sh/hook-weight"] value: "5" + - equal: + path: metadata.annotations["helm.sh/hook-delete-policy"] + value: before-hook-creation,hook-succeededApply the same assertion block to both the Role and RoleBinding tests.
Also applies to: 73-75
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/extra/etcd/tests/cleanup_test.yaml` around lines 46 - 48, The Role and RoleBinding cleanup tests only verify hook weight, so extend both assertion blocks to also check metadata.annotations["helm.sh/hook"] and metadata.annotations["helm.sh/hook-delete-policy"]. Update the existing Role and RoleBinding test cases in cleanup_test.yaml so they explicitly confirm these RBAC objects are post-delete hooks and not normal release resources.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/extra/etcd/templates/hooks/cleanup.yaml`:
- Around line 24-26: The PVC delete permissions in the cleanup Role are too
broad because the `persistentvolumeclaims` rule allows namespace-wide access.
Tighten the RBAC in the cleanup template by adding generated `resourceNames` for
the deterministic `data-etcd-*` PVC names and removing `list` if the cleanup Job
only targets known names; keep the rule name-based and aligned with the existing
least-privilege pattern used for secrets.
---
Nitpick comments:
In `@packages/extra/etcd/tests/cleanup_test.yaml`:
- Around line 46-48: The Role and RoleBinding cleanup tests only verify hook
weight, so extend both assertion blocks to also check
metadata.annotations["helm.sh/hook"] and
metadata.annotations["helm.sh/hook-delete-policy"]. Update the existing Role and
RoleBinding test cases in cleanup_test.yaml so they explicitly confirm these
RBAC objects are post-delete hooks and not normal release resources.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: e38358a6-013f-4344-b4c6-aab85eb481bc
📒 Files selected for processing (3)
packages/extra/etcd/Makefilepackages/extra/etcd/templates/hooks/cleanup.yamlpackages/extra/etcd/tests/cleanup_test.yaml
…ook metadata Address CodeRabbit review on #3170: - Tighten the cleanup Role's PVC rule from a namespace-wide get/list/delete to get/delete by explicit resourceNames (data-etcd-<ordinal>, one per replica), mirroring the least-privilege pattern already used for the TLS secrets. The Job only ever deletes those deterministic names, so list is unnecessary and a regression can no longer reach a sibling workload's PVC. - Extend the Role and RoleBinding unit tests to assert helm.sh/hook and helm.sh/hook-delete-policy (not just the weight), so the RBAC cannot regress into normal release resources, and assert the PVC resourceNames. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
…le delete (#3090) When the etcd tenant module is disabled (etcd: false), the etcd-operator StatefulSet data PVCs (data-etcd-<ordinal>) are never garbage-collected, and the five etcd-*-tls secrets are left orphaned in the tenant namespace: etcd-ca-tls / etcd-client-tls are pre-install hooks this chart marks with helm.sh/resource-policy: keep, and etcd-peer-ca-tls / etcd-server-tls / etcd-peer-tls are cert-manager-issued secrets with no owner reference back to the chart. None are reclaimed by the Helm uninstall. Add a post-delete cleanup Job (modeled on the seaweedfs/monitoring sibling fixes) that deletes the data PVCs by name (one per replica) and the five TLS secrets by name. check-release-name pins the release to "etcd", so every target name is deterministic and no sibling workload shares the data-etcd-* prefix. The Job runs non-root with a read-only root fs, dropped capabilities and seccompProfile RuntimeDefault; its RBAC is scoped to PVC get/list/delete and the five secrets by resourceNames. Deletes use --ignore-not-found and failures are logged but never block teardown. The stuck-HelmRelease finalizer noted in the issue is the platform-wide sharded-helm-controller side-bug tracked separately; this change addresses the PVC and secret orphans the etcd chart owns. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
…ook metadata Address CodeRabbit review on #3170: - Tighten the cleanup Role's PVC rule from a namespace-wide get/list/delete to get/delete by explicit resourceNames (data-etcd-<ordinal>, one per replica), mirroring the least-privilege pattern already used for the TLS secrets. The Job only ever deletes those deterministic names, so list is unnecessary and a regression can no longer reach a sibling workload's PVC. - Extend the Role and RoleBinding unit tests to assert helm.sh/hook and helm.sh/hook-delete-policy (not just the weight), so the RBAC cannot regress into normal release resources, and assert the PVC resourceNames. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
8fb4bfc to
4a06df9
Compare
|
Caution Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted. Error details |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/extra/etcd/templates/hooks/cleanup.yaml`:
- Around line 27-33: The cleanup hook template is allowing an explicit replicas:
0 to render an empty resourceNames list in the cleanup job. Update the hook
logic in the cleanup.yaml template around the .Values.replicas usage so it
clamps the count to at least 1 or emits a safe sentinel name, and keep the fix
localized to the resourceNames generation for the PVC cleanup path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: baba44d9-eb48-4658-9d90-5652929eaec5
📒 Files selected for processing (3)
packages/extra/etcd/Makefilepackages/extra/etcd/templates/hooks/cleanup.yamlpackages/extra/etcd/tests/cleanup_test.yaml
✅ Files skipped from review due to trivial changes (1)
- packages/extra/etcd/Makefile
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/extra/etcd/tests/cleanup_test.yaml
… RBAC Address CodeRabbit review on #3170: the PVC cleanup Role generates its resourceNames from .Values.replicas. values.schema.json sets no minimum, so an explicit replicas: 0 rendered an empty resourceNames list — and an empty resourceNames in an RBAC rule matches every PVC in the namespace (a wildcard), silently undoing the least-privilege scoping the previous commit added. Clamp the count to max(1, replicas) via a shared $replicaCount used by both the Role resourceNames range and the delete command, so at least data-etcd-0 is always listed (it exists for any running cluster and the delete uses --ignore-not-found). Add a regression test asserting replicas: 0 still yields a name-scoped Role and a single-name delete rather than a namespace-wide grant. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
IvanHunters
left a comment
There was a problem hiding this comment.
Review: LGTM with non-blocking notes
Correct, narrowly-scoped post-delete cleanup hook, functionally identical to the merged seaweedfs sibling (#3092). RBAC is scoped by name, cross-namespace/cross-tenant match is impossible. helm unittest passes (26 tests).
Verified
- All deletes are
-n {{ .Release.Namespace }}with a namespaced Role, so cross-namespace match is impossible. PVCs are deleted by explicitresourceNames: data-etcd-<i>; the five secrets by literal name, matching exactly thesecretNamevalues inetcd-cluster.yaml. - The
max 1 (int .Values.replicas)clamp prevents an emptyresourceNames(an empty list would act as a wildcard over all PVCs). Covered by thereplicas: 0test. - The pre-install
keep-policy placeholder secrets (etcd-ca-tls,etcd-client-tlsindatastore.yaml) are why a normal uninstall leaves them behind, which is why the explicit delete is needed. - The Job carries
policy.cozystack.io/allow-to-apiserver: "true", required for egress to the API server under the tenant default-deny policy. PSS-restricted securityContext is complete.
Non-blocking notes
- The PR body says the fix is modeled on merged seaweedfs and monitoring siblings; the monitoring cleanup hook is not merged yet (it is PR #3094). Only seaweedfs is a merged precedent.
- Pre-existing, not touched by this PR:
tests/no-post-upgrade-hook_test.yamlliststemplates/hook/job.yamlandtemplates/version.yaml, which do not exist in the tree, so thosecount: 0asserts pass trivially. Worth cleaning up in a follow-up. - Delete commands use
|| echo WARNINGwithoutset -e, so the Job never fails a teardown on a failed delete. This is a deliberate "never block teardown" choice and matches seaweedfs; it differs from clickhouse (set -eu; exit 1). - Consistency: this PR introduces permanent data-PVC deletion on module disable but is not marked as a breaking change, while the equivalent monitoring PR (#3094) carries the
!breaking-change marker. Consider aligning the marker for symmetry.
Upgrade impact: the hook fires only on etcd-module uninstall (etcd: false), so a normal upgrade that keeps the module changes nothing. The intended breaking change (disabling the etcd module now permanently deletes its data PVCs, per #3090) is documented in the PR body WARNING. Fresh install: hook is inactive on install/upgrade.
Family consistency
- Across the sibling PRs the PVC-deletion strategy is not uniform: etcd deletes by explicit name, monitoring (#3094) by label, seaweedfs by
instancelabel. Each is safe on its own; worth aligning eventually.
…nd route it through cozy-lib.images-registry (#3171) ## What this PR does Fixes #3088. The post-delete cleanup Jobs in the **harbor** and **mariadb** charts hardcoded the cleanup image as `docker.io/clastix/kubectl:v1.32` — a moving Docker Hub tag. This: - **broke reproducibility / was a supply-chain risk** — `v1.32` is mutable and can be re-pushed to a different image; - **could not be satisfied on air-gapped / mirrored clusters** — the hardcoded `docker.io` bypassed the cluster images registry, and it failed at the worst moment: during **uninstall**, when the `post-delete` hook runs and a stuck Job stalls release teardown. ### Changes - **New canonical helper `cozy-lib.image`** (`packages/library/cozy-lib/templates/_cozyconfig.tpl`). Given a registry-relative image path and the global context, it prefixes `cozy-lib.images-registry` when set (`<registry>/<image>`) and returns the image unchanged when empty — **with no leading `/`**, so standard installs still resolve from the default registry. This establishes the previously-undefined "how to reference a cluster image" pattern (`images-registry` was defined but never consumed). - **Digest-pinned image** in both cleanup hooks: `clastix/kubectl:v1.32@sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c` (the immutable multi-arch OCI index for the `v1.32` tag; the same digest already pinned for `postgres-operator`'s webhook hook). - **Renovate keeps the pin fresh.** The repo disables the helm-values manager (`enabledManagers`), and the ref is assembled at render time by the helper rather than being a Dockerfile / go.mod dependency, so the built-in managers can't see it. A `custom.regex` manager over the hook templates tracks `clastix/kubectl` by tag+digest — mirroring the existing `extra/etcd` manager — so the supply-chain pin doesn't silently age. - **Hardened `cozy-lib.images-registry`** against a nil `.Values._cluster` so the helper never crashes the render — relevant precisely because this hook renders during teardown. - **Helm unit tests** in both charts assert the rendered image for three cases: empty registry (default), registry set, and the no-leading-slash edge case. Assertions match by **pattern** (routed + digest-pinned + no leading slash) rather than an exact digest, so a Renovate bump keeps them green. ### Verification - `helm unittest` passes for both charts (harbor: 16, mariadb: 10). - Real `helm template` renders confirmed for both charts: - empty registry → `clastix/kubectl:v1.32@sha256:b9ef…` - `images-registry=registry.internal:5000` → `registry.internal:5000/clastix/kubectl:v1.32@sha256:b9ef…` - The Renovate `matchStrings` regex verified to capture `currentValue=v1.32` and the digest from both hook templates. ### Note on the wider footprint The same `docker.io/clastix/kubectl:v1.32` hardcode is already merged on `main` in several other cleanup hooks (bucket, qdrant, gateway, seaweedfs, kubernetes, tenant, dashboard, keycloak-configure) and appears in the open PRs #3170 (etcd), #3094 (monitoring), #3072 (clickhouse). This PR fixes harbor + mariadb per the issue and establishes the `cozy-lib.image` + Renovate pattern; migrating the remaining hooks is a follow-up. ### Release note ```release-note fix(cozy-lib,harbor,mariadb): digest-pin the harbor and mariadb post-delete cleanup-hook kubectl image and route it through the cluster images registry (`cozy-lib.images-registry`) via the new `cozy-lib.image` helper, so mirrored / air-gapped installs can resolve it and uninstall no longer depends on a moving Docker Hub tag; a custom Renovate manager keeps the digest fresh ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Updated post-delete cleanup hook Jobs to use digest-pinned container images for more secure, repeatable execution. * Improved image reference rendering so cleanup hooks work correctly with or without an image registry configured, avoiding invalid leading/trailing slash issues. * **Tests** * Extended cleanup hook rendering tests to validate digest pinning and correct registry prefix behavior, including edge cases. * **Chores** * Enhanced automated dependency detection to treat digest-pinned hook image references as stable. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
What this PR does
When the
etcdtenant module is disabled (etcd: false), the etcd HelmRelease is pruned but two classes of resources leak into the tenant namespace (reported in #3090):volumeClaimTemplatePVCsdata-etcd-<ordinal>are never garbage-collected onEtcdClusterdeletion.etcd-*-tlssecrets survive the uninstall:etcd-ca-tls/etcd-client-tlsare created by this chart aspre-installhooks withhelm.sh/resource-policy: keep, so Helm deliberately keeps them;etcd-peer-ca-tls/etcd-server-tls/etcd-peer-tlsare populated by cert-manager fromCertificateresources and carry no owner reference back to the chart.This adds a
post-deletecleanup hook (packages/extra/etcd/templates/hooks/cleanup.yaml), modeled on the accepted seaweedfs (#3092) and monitoring (#3091) sibling fixes, that:data-etcd-0 … data-etcd-<replicas-1>);etcd-*-tlssecrets by name.check-release-namepins the release name toetcd, so every target name is deterministic and no sibling workload shares thedata-etcd-*prefix in a tenant namespace — deletion by name is safe. cert-manager TLS secrets carry only the cluster-widecontroller.cert-manager.io/faolabel, so the RBAC grantsdeleteby explicitresourceNamesrather than an unsafe label sweep.The cleanup Job is hardened: non-root, read-only root fs, dropped capabilities,
seccompProfile: RuntimeDefault, resource requests/limits,activeDeadlineSeconds. Deletes use--ignore-not-foundand failures are logged but never block teardown.A Helm unit-test suite (
tests/cleanup_test.yaml) asserts the hook metadata/RBAC/hardening and that the rendered commands target exactly the reported PVCs and secrets, including scaling the PVC list to the configured replica count.Note
The stuck HelmRelease finalizer noted in #3090 is the platform-wide sharded-helm-controller side-bug (tracked separately); it affects every
chartRef-based tenant module identically, not just etcd. This PR fixes the PVC and secret orphans the etcd chart itself owns, matching the scope of the merged seaweedfs/monitoring fixes.Warning
Disabling the etcd module now permanently deletes its data PVCs and the etcd data they hold. This is intentional and fixes the storage leak. Back up before disabling if the data matters.
Fixes #3090.
Release note
Summary by CodeRabbit
testmake target as phony for the etcd package.