fix(kubernetes): bound the remaining pre-delete hook deletions - #3604

What this PR does

Three of the six steps in the kubernetes pre-delete hook call kubectl delete with neither --wait=false nor --timeout. --wait defaults to true and --timeout=0 is implemented as a week, so the KamajiControlPlane delete in step 3, the DataVolume delete in step 5 and the LoadBalancer Service delete in step 6 each park the hook for 168 hours on one stuck finalizer. A DataVolume held by a CDI importer or a PVC finalizer is the reachable case.

The waits in steps 2 and 4 were bounded earlier, in response to a report that named the child-HelmRelease wait and nothing else. The fix stopped where the report did, so the shape it described stayed in the three steps nobody had looked at.

None of the three waits now. Nothing downstream reads the result of steps 5 and 6. What matters after step 3 is the TenantControlPlane, and step 4 deletes that one directly and waits for it, so blocking on the KamajiControlPlane first only adds a second place for a finalizer to stall. An expired wait and --wait=false leave the same cluster state, with the same objects finalizing in the background.

Not waiting also keeps these steps under set -e with no guard, which matters more than it looks. A guard cannot tell an expired wait from a rejected request, so it would report a failed delete as cleanup that happened. With no timeout to expire, every non-zero exit left is a real failure.

The watch verb goes away for those three resources. kubectl opens a watch only on the wait path, so the grant was unexercised. helmreleases and tenantcontrolplanes keep it because steps 2 and 4 still wait, and there the verb is load-bearing in a way that is easy to miss: removing it does not produce an error, it hangs the delete. That is the failure this repo already hit once at the tenant level, so the whole rule set is pinned rather than left to the next reviewer of this Role.

backoffLimit goes to 0, against a default of 6 that restartPolicy: Never turns into seven pods in series, each restarting the script from step 1 and each paying its own image pull.

What this PR does not do

It does not add activeDeadlineSeconds, and that is arithmetic rather than oversight. A deadline has to sit above the sum of the remaining waits, or it cuts runs that were going to finish, and far enough under helm-controller's uninstall budget to be the thing helm reports rather than something helm waits through. Those two bounds do not currently leave a gap, and the image pull comes from Docker Hub with no mirror in front of it. Room comes from raising spec.uninstall.timeout on the generated HelmRelease first; squeezing the waits further is the wrong direction.

Worth stating plainly, because it is the one change here that is not purely a tightening: with backoffLimit: 0 a pod eviction now fails the whole uninstall attempt, where six retries previously absorbed it. That is deliberate. Both remaining waits are guarded, so an attempt can spend its entire budget without failing, and a retry would restart from step 1 with the clock already there; on a nodeless tenant it would also re-pay step 4, since the first attempt leaves the TenantControlPlane still terminating. The failure a retry would cover is an eviction, which is not bounded in time and costs most when it arrives late. Flux still repeats the uninstall on its own interval, so the end state is preserved and the trade is wall-clock for predictability.

It does not bound the requests. --wait=false and --timeout bound the waiting, so an apiserver that accepts the connection and then stops answering hangs this hook exactly as before. That is a different failure from the stuck finalizer, it is not made worse here, and it wants one answer across every call rather than a flag on the deletes.

Tests

packages/apps/kubernetes/tests/delete_hook_test.yaml gains cases for the bounds and for the Role. Every delete in the hook is asserted to carry one, the Job is pinned to a single attempt, and the rule set is pinned whole.

The regex patterns walk only continuation lines, the ones ending in a backslash, so each stays inside the command it names. A cross-line wildcard reaches a flag belonging to the next command instead, and RE2 has no lookahead to exclude the echo in between.

The Role assertions use whole-rule contains plus lengthEqual rather than per-index verb checks. notContains passes when its path does not resolve, so an index-addressed "and nowhere else" goes quiet the moment the rules list is reordered or shortened, which is exactly the edit it exists to catch. Matching the entire rule pins the verb set in both directions and names the resource when it fails.

Every pinned value was mutation-tested: dropping --wait=false from each of the three deletes, dropping --timeout from either of the two that still wait, backoffLimit in both directions, adding watch back, removing watch from a resource that waits, appending a rule past everything asserted, and reordering the rules list. Each one fails the assertion that claims that ground and no other.

Screenshots

None. This changes a Job manifest and its tests, with no user-visible surface.

Downstream repositories

Walked the trigger map against the diff. The diff is one Job manifest inside the kubernetes chart and its helm-unittest suite. No package is added, renamed or removed; no values.schema.json, values.yaml default or version enum changes; no ApplicationDefinition semantics, namespace, variant or release asset changes; nothing under hack/ is touched and no make target changes behaviour.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:

Release note

fix(kubernetes): bound the pre-delete hook's remaining unbounded deletions and limit its Job to a single attempt, so a stuck finalizer on a DataVolume, LoadBalancer Service or KamajiControlPlane no longer parks the hook and blocks the uninstall

Summary by CodeRabbit

• Bug Fixes
  • Improved Kubernetes resource cleanup during deletion to avoid indefinite waits.
  • Cleanup operations now complete with bounded waiting and fewer retry attempts, improving reliability and responsiveness.
  • Removed unnecessary access permissions from cleanup operations, strengthening security.
• Tests
  • Added coverage to verify deletion timeouts, retry behavior, and required permissions across supported resources.