Skip to content

fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry - #3171

Merged
scooby87 merged 1 commit into
mainfrom
fix/3088-cleanup-image-digest-registry
Jul 2, 2026
Merged

fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry#3171
scooby87 merged 1 commit into
mainfrom
fix/3088-cleanup-image-digest-registry

Conversation

@scooby87

@scooby87 scooby87 commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Fixes #3088.

The post-delete cleanup Jobs in the harbor and mariadb charts hardcoded the cleanup image as docker.io/clastix/kubectl:v1.32 — a moving Docker Hub tag. This:

  • broke reproducibility / was a supply-chain riskv1.32 is mutable and can be re-pushed to a different image;
  • could not be satisfied on air-gapped / mirrored clusters — the hardcoded docker.io bypassed the cluster images registry, and it failed at the worst moment: during uninstall, when the post-delete hook runs and a stuck Job stalls release teardown.

Changes

  • New canonical helper cozy-lib.image (packages/library/cozy-lib/templates/_cozyconfig.tpl). Given a registry-relative image path and the global context, it prefixes cozy-lib.images-registry when set (<registry>/<image>) and returns the image unchanged when empty — with no leading /, so standard installs still resolve from the default registry. This establishes the previously-undefined "how to reference a cluster image" pattern (images-registry was defined but never consumed).
  • Digest-pinned image in both cleanup hooks: clastix/kubectl:v1.32@sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c (the immutable multi-arch OCI index for the v1.32 tag; the same digest already pinned for postgres-operator's webhook hook).
  • Renovate keeps the pin fresh. The repo disables the helm-values manager (enabledManagers), and the ref is assembled at render time by the helper rather than being a Dockerfile / go.mod dependency, so the built-in managers can't see it. A custom.regex manager over the hook templates tracks clastix/kubectl by tag+digest — mirroring the existing extra/etcd manager — so the supply-chain pin doesn't silently age.
  • Hardened cozy-lib.images-registry against a nil .Values._cluster so the helper never crashes the render — relevant precisely because this hook renders during teardown.
  • Helm unit tests in both charts assert the rendered image for three cases: empty registry (default), registry set, and the no-leading-slash edge case. Assertions match by pattern (routed + digest-pinned + no leading slash) rather than an exact digest, so a Renovate bump keeps them green.

Verification

  • helm unittest passes for both charts (harbor: 16, mariadb: 10).
  • Real helm template renders confirmed for both charts:
    • empty registry → clastix/kubectl:v1.32@sha256:b9ef…
    • images-registry=registry.internal:5000registry.internal:5000/clastix/kubectl:v1.32@sha256:b9ef…
  • The Renovate matchStrings regex verified to capture currentValue=v1.32 and the digest from both hook templates.

Note on the wider footprint

The same docker.io/clastix/kubectl:v1.32 hardcode is already merged on main in several other cleanup hooks (bucket, qdrant, gateway, seaweedfs, kubernetes, tenant, dashboard, keycloak-configure) and appears in the open PRs #3170 (etcd), #3094 (monitoring), #3072 (clickhouse). This PR fixes harbor + mariadb per the issue and establishes the cozy-lib.image + Renovate pattern; migrating the remaining hooks is a follow-up.

Release note

fix(cozy-lib,harbor,mariadb): digest-pin the harbor and mariadb post-delete cleanup-hook kubectl image and route it through the cluster images registry (`cozy-lib.images-registry`) via the new `cozy-lib.image` helper, so mirrored / air-gapped installs can resolve it and uninstall no longer depends on a moving Docker Hub tag; a custom Renovate manager keeps the digest fresh

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Updated post-delete cleanup hook Jobs to use digest-pinned container images for more secure, repeatable execution.
    • Improved image reference rendering so cleanup hooks work correctly with or without an image registry configured, avoiding invalid leading/trailing slash issues.
  • Tests
    • Extended cleanup hook rendering tests to validate digest pinning and correct registry prefix behavior, including edge cases.
  • Chores
    • Enhanced automated dependency detection to treat digest-pinned hook image references as stable.

@coderabbitai

coderabbitai Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 28c84f5b-75a4-4298-9f0f-6953b535f130

📥 Commits

Reviewing files that changed from the base of the PR and between 123be9e and a76c592.

📒 Files selected for processing (6)
  • .github/renovate.json
  • packages/apps/harbor/templates/hooks/cleanup.yaml
  • packages/apps/harbor/tests/cleanup_hook_test.yaml
  • packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml
  • packages/apps/mariadb/tests/cleanup_pvc_hook_test.yaml
  • packages/library/cozy-lib/templates/_cozyconfig.tpl
✅ Files skipped from review due to trivial changes (1)
  • .github/renovate.json
🚧 Files skipped from review as they are similar to previous changes (3)
  • packages/apps/harbor/templates/hooks/cleanup.yaml
  • packages/library/cozy-lib/templates/_cozyconfig.tpl
  • packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml

📝 Walkthrough

Walkthrough

Harbor and MariaDB cleanup hooks now render digest-pinned clastix/kubectl images through cozy-lib.image. The registry helper is null-safe when _cluster is missing, cleanup-hook tests cover registry prefixing and empty defaults, and Renovate is configured to track the pinned references.

Changes

Digest-pinned cleanup images

Layer / File(s) Summary
Null-safe images-registry helper
packages/library/cozy-lib/templates/_cozyconfig.tpl
cozy-lib.images-registry now defaults .Values._cluster to an empty dict before indexing images-registry, avoiding failure when _cluster is absent.
Harbor cleanup image pinning and tests
packages/apps/harbor/templates/hooks/cleanup.yaml, packages/apps/harbor/tests/cleanup_hook_test.yaml
The Harbor cleanup hook renders a digest-pinned kubectl image through cozy-lib.image, and its tests assert digest pinning, registry prefixing, and empty-registry rendering.
MariaDB cleanup image pinning and tests
packages/apps/mariadb/templates/hooks/cleanup-pvc.yaml, packages/apps/mariadb/tests/cleanup_pvc_hook_test.yaml
The MariaDB cleanup hook renders a digest-pinned kubectl image through cozy-lib.image, and its tests assert Job metadata plus digest pinning, registry prefixing, and empty-registry rendering.
Renovate kubectl manager
.github/renovate.json
A custom Renovate manager is added to detect digest-pinned clastix/kubectl references in hook templates.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested labels: kind/cleanup

Suggested reviewers: kvaps, lllamnyp, lexfrei

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately summarizes the main change: digest-pinning cleanup-hook kubectl images and routing them through cozy-lib.
Linked Issues check ✅ Passed The Harbor and MariaDB cleanup hooks now use digest-pinned images via the registry-aware helper, and tests cover registry and empty-default rendering.
Out of Scope Changes check ✅ Passed The helper null-safety fix and Renovate matcher support the same image-pinning flow and do not appear unrelated to the issue scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/3088-cleanup-image-digest-registry

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses reliability and security concerns in the Harbor and MariaDB cleanup hooks. By replacing hardcoded, mutable image tags with digest-pinned references and introducing a flexible registry routing helper, the changes ensure that cleanup jobs remain functional in air-gapped clusters and maintain consistent image resolution across different deployment environments.

Highlights

  • New Image Helper: Introduced a canonical cozy-lib.image helper to dynamically route container images through the cluster's configured images-registry.
  • Digest Pinning: Updated cleanup hooks in Harbor and MariaDB to use digest-pinned images instead of mutable tags, improving supply-chain security and reproducibility.
  • Air-Gapped Support: Ensured cleanup hooks function correctly in air-gapped or mirrored environments by allowing registry prefixing while preventing invalid leading slashes.
  • Robustness: Hardened cozy-lib.images-registry to handle cases where .Values._cluster is nil, preventing render failures during teardown.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@github-actions github-actions Bot added size/L This PR changes 100-499 lines, ignoring generated files area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) labels Jul 1, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new helper template cozy-lib.image to build fully-qualified image references routed through the cluster images registry, and updates the cleanup hooks in harbor and mariadb to use this helper with digest-pinned images. It also adds corresponding tests to verify image routing and registry prefixing. The reviewer suggested trimming potential leading/trailing slashes from the image path and registry to prevent malformed image references with double slashes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +42 to +45
{{- $image := index . 0 }}
{{- $ctx := index . 1 }}
{{- $registry := include "cozy-lib.images-registry" $ctx }}
{{- with $registry }}{{ . }}/{{ end }}{{ $image }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To prevent potential malformed image references with double slashes (e.g., if images-registry is configured with a trailing slash or if the image path starts with a leading slash), it is safer to trim any trailing slash from the registry and any leading slash from the image path.

{{- $image := index . 0 | trimPrefix "/" }}
{{- $ctx := index . 1 }}
{{- $registry := include "cozy-lib.images-registry" $ctx | trimSuffix "/" }}
{{- with $registry }}{{ . }}/{{ end }}{{ $image }}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — applied in a76c592. The cozy-lib.image helper now trimPrefix "/" on the image path and trimSuffix "/" on the registry, so a misconfigured images-registry with a trailing slash (e.g. registry.example.com/) no longer yields a double slash. Added a regression test in both the harbor and mariadb cleanup-hook suites asserting the rendered ref contains no //. Thanks!

@scooby87
scooby87 force-pushed the fix/3088-cleanup-image-digest-registry branch 2 times, most recently from 57bd941 to 123be9e Compare July 2, 2026 13:40
…nd route it through cozy-lib.images-registry

The post-delete cleanup Jobs in the harbor and mariadb charts hardcoded
`docker.io/clastix/kubectl:v1.32` — a moving tag pulled directly from
Docker Hub. This broke build reproducibility (the tag can be re-pushed),
was a supply-chain risk, and could not be satisfied on air-gapped /
mirrored clusters, where it failed at the worst moment: during uninstall,
when the post-delete hook runs and stalls release teardown.

Add a canonical `cozy-lib.image` helper that prefixes a registry-relative
image path with `cozy-lib.images-registry` when set and returns it
unchanged (no leading "/") when empty, so standard installs resolve from
the default registry and mirrored installs resolve internally. Consume it
from both cleanup hooks with the image digest-pinned to the immutable
multi-arch index sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c
(the same digest already pinned for postgres-operator's webhook hook).

Keep the pin fresh with a custom Renovate manager: the repo disables the
helm-values manager (enabledManagers), and the ref is built at render time
by the helper rather than being a Dockerfile/go.mod dependency, so the
built-in managers cannot see it. A regex manager over the hook templates
tracks clastix/kubectl by tag+digest — mirroring the existing extra/etcd
manager — so the supply-chain pin does not silently age.

Also harden `cozy-lib.images-registry` against a nil .Values._cluster so
the helper never crashes the render — relevant precisely because the hook
renders during teardown. Helm unit tests assert the rendered image (routed,
digest-pinned, no leading slash) by pattern rather than an exact digest, so
a Renovate bump keeps them green.

Closes #3088

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
@scooby87
scooby87 force-pushed the fix/3088-cleanup-image-digest-registry branch from 123be9e to a76c592 Compare July 2, 2026 14:45

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: LGTM with non-blocking notes

The digest-pin and the cozy-lib.image routing are correct across all three charts. Both chart unittests pass (harbor 16, mariadb 10).

Main note (accuracy of framing)

  • cozy-lib.images-registry reads .Values._cluster["images-registry"], but that key is never populated anywhere in the tree. The cozystack-values emitter (packages/core/platform/templates/apps.yaml) emits ~40 _cluster keys and images-registry is not one of them; there is no Go injector and no installer manifest that sets it. So on every real cluster today the registry resolves empty and cozy-lib.image returns the image unchanged. The routing is correct and future-proof, but it is currently dormant: air-gapped resolution is not achieved until a separate change populates _cluster.images-registry. Suggest softening the PR body / release-note to "establishes the routing" rather than implying present air-gapped capability.

Verified

  • helm template renders clastix/kubectl:v1.32@sha256:b9ef7d8… with empty registry (no leading slash) and registry.example.com/clastix/kubectl:v1.32@sha256:b9ef7d8… with the registry set.
  • nil-_cluster hardening ((.Values._cluster | default dict)) is necessary: the old index .Values._cluster "images-registry" fails with index of untyped nil when _cluster is unset.
  • Bare clastix/kubectl (no docker.io/ prefix) resolves to docker.io/clastix/kubectl, so the join produces <mirror>/clastix/kubectl rather than <mirror>/docker.io/clastix/kubectl.
  • Digest matches the pin already at packages/system/postgres-operator/values.yaml.
  • Renovate manager matchStrings requires @sha256: and managerFilePatterns matches both changed files.

Follow-ups (separate PR, as the body acknowledges)

  • Migrate the remaining hardcoded docker.io/clastix/kubectl:v1.32 cleanup hooks to the same pattern (keycloak-configure, dashboard, bucket, qdrant, kubernetes, seaweedfs, gateway, clickhouse, tenant).
  • Some of those are not under a templates/hooks/ directory, so they are not caught by this PR's Renovate managerFilePatterns; broaden the pattern when migrating them or the pins will silently age.

Upgrade impact: none (hooks fire only on post-delete; rendered image is registry-identical to before plus an immutable digest). Fresh install: none negative.

@scooby87
scooby87 enabled auto-merge July 2, 2026 15:21
@scooby87
scooby87 merged commit e4c6802 into main Jul 2, 2026
75 of 76 checks passed
@scooby87
scooby87 deleted the fix/3088-cleanup-image-digest-registry branch July 2, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/database Issues or PRs related to managed databases (postgres, mariadb, redis, etcd, kafka, clickhouse) area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(images): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry (harbor, mariadb)

2 participants