fix(backups): carry dropdown option sources in CRD annotations, not schema - #2823
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughMoves CozyStack CRD dropdown source metadata from per-field ChangesCRD source metadata annotation migration
Sequence Diagram(s)sequenceDiagram
participant Dev as Developer (add markers)
participant GoTypes as Go API types
participant ControllerGen as controller-gen
participant CRDYAML as Generated CRD YAMLs
participant Tests as Helm unit tests
Dev->>GoTypes: add kubebuilder:metadata:annotations markers
GoTypes->>ControllerGen: controller-gen reads markers
ControllerGen->>CRDYAML: emit metadata.annotations (options.cozystack.io/source.*)
Tests->>CRDYAML: validate annotations present and x-cozystack-options absent
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related issues
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…chema
The four backups.cozystack.io CRDs injected the x-cozystack-options vendor
extension into their OpenAPI validation schema. apiextensions JSONSchemaProps
is a closed struct that only preserves x-kubernetes-* vendor extensions, so
helm-controller's server-side apply rejected the unknown key ("field not
declared in schema") and the CRDs never applied — the controller crash-looped
and backups were non-functional.
Move the field-to-source mapping the dashboard needs into CRD
metadata.annotations (preserved by the apiserver), emitted via
+kubebuilder:metadata:annotations markers, and drop the post-processing awk
injector. The dashboard reattaches the extension client-side.
A helm-unittest suite pins the contract: each dropdown field carries its source
in metadata.annotations and must not carry x-cozystack-options in the schema,
guarding against re-introducing the injector.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
f0e959d to
2363f2d
Compare
82ed4e2 to
6901715
Compare
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request resolves a critical issue where the cozy-backup-controller failed to reconcile due to invalid vendor extensions in CRD schemas. By moving these mappings to metadata annotations and cleaning up the injection process, the CRDs are now correctly accepted by the apiserver. Additionally, the PR strengthens the CI pipeline by improving how HelmRelease readiness is verified, ensuring that deployment failures are caught and reported clearly. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request refactors how dropdown option sources are exposed in the backup-controller CRDs. Instead of injecting the custom x-cozystack-options vendor extension directly into the OpenAPI validation schema (which is rejected by server-side apply), the sources are now defined via metadata.annotations on the CRDs. The awk-based injection script (hack/inject-cozystack-options.awk) has been removed, and the annotations are now generated directly from kubebuilder markers in the Go API types. Additionally, a Helm unittest suite has been added to verify this contract, and the E2E installation script (hack/e2e-install-cozystack.bats) has been updated to improve HelmRelease readiness checks. Feedback on the E2E script highlights potential vulnerabilities to transient kubectl failures and false positives in the readiness check loop, suggesting a more robust check using custom columns.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if ! timeout 120 sh -ec 'while kubectl get hr -A --no-headers | grep -v " True " | grep -q .; do sleep 5; done'; then | ||
| kubectl get hr -A | ||
| # kubectl's STATUS column truncates long messages; dump the full Ready | ||
| # condition per non-ready HR so the real error (e.g. a rejected CRD) is | ||
| # visible in the test output instead of only inside the cozyreport. | ||
| kubectl get hr -A --no-headers | grep -v " True " | while read -r ns name rest; do | ||
| echo "--- Non-ready HelmRelease: $ns/$name" >&2 | ||
| kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2 | ||
| done |
There was a problem hiding this comment.
There are two critical issues with the current readiness check implementation:
- Transient
kubectlFailure Vulnerability: The loop conditionwhile kubectl get hr -A --no-headers | grep -v " True " | grep -q .; do sleep 5; donewill immediately terminate and succeed ifkubectlfails transiently (e.g., due to rate limits or temporary API server unavailability). This is because a failure inkubectlresults in empty output, which causesgrep -q .to exit with1(false), ending the loop and letting the test pass silently. - False Positives from Status Messages: Grepping the entire line for
" True "can lead to false positives if a non-ready HelmRelease has a status message or description containing the substring" True "(e.g.,"reconciliation set to True"or"dependency True is not ready").
Solution
- Store the
kubectloutput in a variable and only evaluate readiness if the command succeeded. - Use
custom-columnsto place theReadycondition status as the first column, and then usegrep -v "^True"to precisely filter out ready releases, completely avoiding false matches in the namespace, name, or status message.
if ! timeout 120 sh -ec '
while true; do
if out=$(kubectl get hr -A -o custom-columns=READY:.status.conditions[?(@.type=="Ready")].status,NS:.metadata.namespace,NAME:.metadata.name --no-headers 2>/dev/null); then
if ! printf "%s\n" "$out" | grep -v "^True" | grep -q .; then
exit 0
fi
fi
sleep 5
done
'; then
kubectl get hr -A
# kubectl's STATUS column truncates long messages; dump the full Ready
# condition per non-ready HR so the real error (e.g. a rejected CRD) is
# visible in the test output instead of only inside the cozyreport.
kubectl get hr -A -o custom-columns=READY:.status.conditions[?(@.type=="Ready")].status,NS:.metadata.namespace,NAME:.metadata.name --no-headers | grep -v "^True" | while read -r status ns name; do
echo "--- Non-ready HelmRelease: $ns/$name (status: $status)" >&2
kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2
done
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/system/backup-controller/tests/crds-option-sources_test.yaml (1)
20-82: ⚡ Quick winWell-structured test assertions validate the annotation contract comprehensively.
The four test cases correctly validate that:
- Each CRD carries the required
options.cozystack.io/source.*annotations in metadata- No
x-cozystack-optionsvendor extensions remain in the OpenAPI schemaThe assertion paths accurately navigate the CRD structure, and the expected annotation keys and values match the actual CRD definitions in the context snippets.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/backup-controller/tests/crds-option-sources_test.yaml` around lines 20 - 82, The test assertions correctly verify presence of metadata annotations (e.g. metadata.annotations["options.cozystack.io/source.applicationRef.kind"], metadata.annotations["options.cozystack.io/source.planRef.name"], metadata.annotations["options.cozystack.io/source.backupClassName"], metadata.annotations["options.cozystack.io/source.backupRef.name"], metadata.annotations["options.cozystack.io/source.targetApplicationRef.kind"]) and absence of vendor extensions in the OpenAPI schema (e.g. spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.applicationRef.properties.kind["x-cozystack-options"], spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.planRef.properties.name["x-cozystack-options"], spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.backupClassName["x-cozystack-options"], spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.backupRef.properties.name["x-cozystack-options"], spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.targetApplicationRef.properties.kind["x-cozystack-options"]); no code changes are required—keep the assertions as-is, or if you want extra safety, add one additional assertion per CRD to check the annotation keys exist before validating values (use the same metadata.annotations[...] paths) to make failures clearer.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/e2e-install-cozystack.bats`:
- Line 130: The pipeline in the while condition (the command starting with
"kubectl get hr -A --no-headers | grep -v \" True \" | grep -q .") can hide
kubectl failures under POSIX sh because pipeline exit status isn't propagated;
change the loop so you capture kubectl's exit and output explicitly and only
consider the grep result if kubectl succeeded. Concretely, replace the one-liner
with a loop that runs "kubectl get hr -A --no-headers" into a temporary buffer
(or variable), check kubectl's exit status, and then run "grep -v ' True '" and
"grep -q ." against that buffer (sleeping and retrying on either a non-empty
grep result or a transient kubectl failure) so the timeout branch triggers
correctly on kubectl errors.
---
Nitpick comments:
In `@packages/system/backup-controller/tests/crds-option-sources_test.yaml`:
- Around line 20-82: The test assertions correctly verify presence of metadata
annotations (e.g.
metadata.annotations["options.cozystack.io/source.applicationRef.kind"],
metadata.annotations["options.cozystack.io/source.planRef.name"],
metadata.annotations["options.cozystack.io/source.backupClassName"],
metadata.annotations["options.cozystack.io/source.backupRef.name"],
metadata.annotations["options.cozystack.io/source.targetApplicationRef.kind"])
and absence of vendor extensions in the OpenAPI schema (e.g.
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.applicationRef.properties.kind["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.planRef.properties.name["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.backupClassName["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.backupRef.properties.name["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.targetApplicationRef.properties.kind["x-cozystack-options"]);
no code changes are required—keep the assertions as-is, or if you want extra
safety, add one additional assertion per CRD to check the annotation keys exist
before validating values (use the same metadata.annotations[...] paths) to make
failures clearer.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: da1b84a6-3b57-4674-8fb2-6c608f6bdc4c
📒 Files selected for processing (13)
api/backups/v1alpha1/backup_types.goapi/backups/v1alpha1/backupjob_types.goapi/backups/v1alpha1/plan_types.goapi/backups/v1alpha1/restorejob_types.gohack/e2e-install-cozystack.batshack/inject-cozystack-options.awkhack/update-codegen.shpackages/system/backup-controller/Makefilepackages/system/backup-controller/definitions/backups.cozystack.io_backupjobs.yamlpackages/system/backup-controller/definitions/backups.cozystack.io_backups.yamlpackages/system/backup-controller/definitions/backups.cozystack.io_plans.yamlpackages/system/backup-controller/definitions/backups.cozystack.io_restorejobs.yamlpackages/system/backup-controller/tests/crds-option-sources_test.yaml
💤 Files with no reviewable changes (2)
- hack/inject-cozystack-options.awk
- hack/update-codegen.sh
Andrei Kvapil (kvaps)
left a comment
There was a problem hiding this comment.
Verified the mapping parity against the old awk targets (all four CRDs, field→source pairs match 1:1), the annotation values against the provider names in pkg/registry/core/option/providers.go, and that the codegen-drift check proves the markers reproduce the committed YAML. The unittest suite pinning the contract in both directions is a nice touch.
Two notes on the e2e gate:
- The authoritative re-list can pass vacuously: in
kubectl get hr -A --no-headers | grep -v " True " | grep -q ., a failed kubectl call (or empty output) makes the pipeline return 1, the loop exits, and the gate passes — POSIXsh -ecdoesn't catch pipeline-internal failures. Worth asserting that kubectl succeeded and the HR list is non-empty (it's guaranteed >0 at this point). - Pre-existing nit:
grep -v " True "matches anywhere in the line, so an HR whose STATUS message contains " True " would be hidden;awk '$4 != "True"'on the READY column would be exact.
Neither is blocking. Since the fix is precisely "HR fails to apply", a full e2e run on this PR before merge would be the real proof — and it exercises the restored gate too.
myasnikovdaniil
left a comment
There was a problem hiding this comment.
Took another pass at the restored e2e gate. Both bots correctly spotted that the readiness loop can still pass green on a kubectl error, but their fixes keep the hand-rolled loop. Proposing the kubectl wait idiom this file already uses everywhere else instead — details inline.
| # Fail the test if any HelmRelease is not Ready. Re-list rather than trust | ||
| # the wait above so late-created HRs are gated too; the brief retry absorbs | ||
| # momentary Unknown flaps from helm-controller drift reconciles. | ||
| if ! timeout 120 sh -ec 'while kubectl get hr -A --no-headers | grep -v " True " | grep -q .; do sleep 5; done'; then | ||
| kubectl get hr -A | ||
| # kubectl's STATUS column truncates long messages; dump the full Ready | ||
| # condition per non-ready HR so the real error (e.g. a rejected CRD) is | ||
| # visible in the test output instead of only inside the cozyreport. | ||
| kubectl get hr -A --no-headers | grep -v " True " | while read -r ns name rest; do | ||
| echo "--- Non-ready HelmRelease: $ns/$name" >&2 | ||
| kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2 | ||
| done | ||
| echo "Some HelmReleases failed to reconcile" >&2 | ||
| exit 1 | ||
| fi |
There was a problem hiding this comment.
Both bots flagged the hole here and they're right, but their fixes keep the hand-rolled loop. No shell option can rescue this shape:
- a kubectl/API error → empty stdout →
grep -q .exits 1 → thewhilecondition is false → the loop ends andshexits 0 → the gate passes green. The exact masking class this PR set out to fix. pipefailcan't help even where available (shis dash on the runners): kubectl's error exit is1andgrep -q's no-match exit is1— indistinguishable. The verdict must not come out of a pipeline at all.grep " True "also substring-matches the free-text STATUS message, so a non-ready HR whose message containsTruereads as ready.
Everywhere else in this file the readiness gate is kubectl wait ... --for=condition=ready (L153, L201, L277, …). The same idiom works here and makes every failure mode a non-zero exit: non-ready HR (named in the output), API error, even zero HRs matched (no matching resources found). Unknown flaps are absorbed natively — wait waits for the condition to become true within the window, which is what the 120s retry loop was hand-rolling.
| # Fail the test if any HelmRelease is not Ready. Re-list rather than trust | |
| # the wait above so late-created HRs are gated too; the brief retry absorbs | |
| # momentary Unknown flaps from helm-controller drift reconciles. | |
| if ! timeout 120 sh -ec 'while kubectl get hr -A --no-headers | grep -v " True " | grep -q .; do sleep 5; done'; then | |
| kubectl get hr -A | |
| # kubectl's STATUS column truncates long messages; dump the full Ready | |
| # condition per non-ready HR so the real error (e.g. a rejected CRD) is | |
| # visible in the test output instead of only inside the cozyreport. | |
| kubectl get hr -A --no-headers | grep -v " True " | while read -r ns name rest; do | |
| echo "--- Non-ready HelmRelease: $ns/$name" >&2 | |
| kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2 | |
| done | |
| echo "Some HelmReleases failed to reconcile" >&2 | |
| exit 1 | |
| fi | |
| # Fail the test if any HelmRelease is not Ready. Wait again on a fresh | |
| # listing so HelmReleases created after the snapshot above are gated too; | |
| # the window absorbs momentary Unknown flaps from drift reconciles. | |
| if ! kubectl wait hr --all -A --timeout=2m --for=condition=ready; then | |
| kubectl get hr -A || true | |
| # kubectl's STATUS column truncates long messages; dump the full Ready | |
| # condition per non-ready HR so the real error (e.g. a rejected CRD) is | |
| # visible in the test output instead of only inside the cozyreport. | |
| kubectl get hr -A --no-headers | awk '$4 != "True"' | while read -r ns name _; do | |
| echo "--- Non-ready HelmRelease: $ns/$name" >&2 | |
| kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2 || true | |
| done | |
| echo "Some HelmReleases failed to reconcile" >&2 | |
| exit 1 | |
| fi |
Notes:
awk '$4 != "True"'keys on the READY column positionally instead of substring-matching the whole line.- the
|| trues live only in the failure path: bats aborts on any command error, so the diagnostics stay best-effort while theexit 1verdict is unconditional.
There was a problem hiding this comment.
Applied verbatim and amended into the e2e commit — the repo merges without squash, so I kept the history at two commits rather than stacking a fixup. Agreed the verdict must not come out of a pipeline: routing it through kubectl wait's own exit code makes a non-ready HR, an API error, and zero matches all non-zero — exactly the masking class this change set out to close. This also folds in the other readiness-check notes on the thread (substring match on the STATUS column, now awk '$4 != "True"' on READY).
| # Pacing only (|| true): a backgrounded fan-out's exit codes are discarded | ||
| # by a bare POSIX `wait`, which let permanently-failing HRs ship green | ||
| # (#2822). A single kubectl wait names every HR that timed out in the | ||
| # trace; the authoritative pass/fail gate is the outcome-based re-list | ||
| # below, which also covers HRs created after this snapshot. | ||
| kubectl wait hr --all -A --timeout=15m --for=condition=ready || true |
There was a problem hiding this comment.
With the gate below becoming a plain kubectl wait, this comment can shrink — the fan-out history lives in #2822.
| # Pacing only (|| true): a backgrounded fan-out's exit codes are discarded | |
| # by a bare POSIX `wait`, which let permanently-failing HRs ship green | |
| # (#2822). A single kubectl wait names every HR that timed out in the | |
| # trace; the authoritative pass/fail gate is the outcome-based re-list | |
| # below, which also covers HRs created after this snapshot. | |
| kubectl wait hr --all -A --timeout=15m --for=condition=ready || true | |
| # Pacing only: names every HR that timed out in the trace; the authoritative | |
| # gate re-lists below, covering HRs created after this snapshot (#2822). | |
| kubectl wait hr --all -A --timeout=15m --for=condition=ready || true |
There was a problem hiding this comment.
Applied.
6901715 to
6039330
Compare
The HR readiness gate in e2e-install-cozystack.bats was toothless in two layers: the backgrounded kubectl-wait fan-out discarded child exit codes (POSIX `wait` without args returns 0), and the final check echoed "Some HelmReleases failed to reconcile" without exiting non-zero — a regression from 1f24038, which replaced the unavailable bats `fail` helper with a bare echo. A permanently-failing platform HelmRelease (e.g. the backup-controller CRD rejection, #2822) shipped through green CI for weeks. Replace the fan-out with a single `kubectl wait hr --all -A` for trace visibility, gate on an outcome-based re-list (which also covers HRs created after the snapshot, with a 120s retry to absorb momentary drift-reconcile flaps), and dump the full Ready condition message per non-ready HR so the real error is visible in the test output. Refs #2822 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
6039330 to
67b1a96
Compare
myasnikovdaniil
left a comment
There was a problem hiding this comment.
The e2e gate now uses the file's kubectl wait idiom — every failure mode (non-ready HR, API error, zero HRs matched) exits non-zero, so a permanently-failing HelmRelease can't ship green again. CRD annotation contract is pinned by the helm-unittest suite, and the option sources all resolve against the providers in pkg/registry/core/option/providers.go. LGTM.
|
Successfully created backport PR for |
What this PR does
Two changes that together fix the v1.5.0 release blocker where the
cozy-backup-controllerHelmRelease failed on every reconcile.Root cause — the four
backups.cozystack.ioCRDs (backupjobs,backups,plans,restorejobs) embedded thex-cozystack-optionsvendor extension directly in their OpenAPI validation schema.apiextensionsJSONSchemaPropsonly preserves the documentedx-kubernetes-*vendor extensions — every otherx-key is rejected by helm-controller's server-side apply (field not declared in schema) and dropped by the apiserver on decode. So the CRDs never applied and the controller crash-looped.This moves the field→source mapping the dashboard needs out of the schema and into CRD
metadata.annotations(which the apiserver preserves), emitted declaratively via+kubebuilder:metadata:annotationsmarkers on the Go types, and removes the post-processing awk injector (hack/inject-cozystack-options.awk). A helm-unittest suite pins the contract: each dropdown field carries its source in annotations and must not carryx-cozystack-optionsin the schema. The dashboard reattaches the extension client-side from these annotations.Annotation contract:
options.cozystack.io/source.<spec-relative-path>: <option-source>, e.g.options.cozystack.io/source.applicationRef.kind: appkind.Detection gap — the install e2e let this ship green: the fanned-out
kubectl waitdiscarded child exit codes (a bare POSIXwaitreturns 0) and the readiness check printed offending HelmReleases without ever exiting non-zero. The gate is now restored — a singlekubectl wait hr --all -Afor trace visibility, an outcome-based re-list that also gates late-created HelmReleases (with a short retry to absorb momentary drift-reconcile flaps) andexit 1s, plus a full Ready-condition dump per non-ready HelmRelease so the real failure reason is visible. This commit is by myasnikovdaniil, consolidated here from #2824 so the root cause and the CI gap land in one PR. With the gate restored, CI onmainstays red until the backups fix in this PR merges — intended.Closes #2822 · Dashboard counterpart: cozystack/cozystack-ui#40
Release note
Summary by CodeRabbit
New Features
Bug Fixes
Refactor