Skip to content

fix(backups): carry dropdown option sources in CRD annotations, not schema - #2823

Merged
Aleksei Sviridkin (lexfrei) merged 2 commits into
mainfrom
fix/backup-controller-crd-options
Jun 4, 2026
Merged

fix(backups): carry dropdown option sources in CRD annotations, not schema#2823
Aleksei Sviridkin (lexfrei) merged 2 commits into
mainfrom
fix/backup-controller-crd-options

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Two changes that together fix the v1.5.0 release blocker where the cozy-backup-controller HelmRelease failed on every reconcile.

Root cause — the four backups.cozystack.io CRDs (backupjobs, backups, plans, restorejobs) embedded the x-cozystack-options vendor extension directly in their OpenAPI validation schema. apiextensions JSONSchemaProps only preserves the documented x-kubernetes-* vendor extensions — every other x- key is rejected by helm-controller's server-side apply (field not declared in schema) and dropped by the apiserver on decode. So the CRDs never applied and the controller crash-looped.

This moves the field→source mapping the dashboard needs out of the schema and into CRD metadata.annotations (which the apiserver preserves), emitted declaratively via +kubebuilder:metadata:annotations markers on the Go types, and removes the post-processing awk injector (hack/inject-cozystack-options.awk). A helm-unittest suite pins the contract: each dropdown field carries its source in annotations and must not carry x-cozystack-options in the schema. The dashboard reattaches the extension client-side from these annotations.

Annotation contract: options.cozystack.io/source.<spec-relative-path>: <option-source>, e.g. options.cozystack.io/source.applicationRef.kind: appkind.

Detection gap — the install e2e let this ship green: the fanned-out kubectl wait discarded child exit codes (a bare POSIX wait returns 0) and the readiness check printed offending HelmReleases without ever exiting non-zero. The gate is now restored — a single kubectl wait hr --all -A for trace visibility, an outcome-based re-list that also gates late-created HelmReleases (with a short retry to absorb momentary drift-reconcile flaps) and exit 1s, plus a full Ready-condition dump per non-ready HelmRelease so the real failure reason is visible. This commit is by myasnikovdaniil, consolidated here from #2824 so the root cause and the CI gap land in one PR. With the gate restored, CI on main stays red until the backups fix in this PR merges — intended.

Closes #2822 · Dashboard counterpart: cozystack/cozystack-ui#40

Release note

fix(backups): fix backup-controller CRDs failing to apply — the dashboard dropdown hint was placed in the CRD schema where apiextensions rejects it; it now lives in CRD annotations

Summary by CodeRabbit

  • New Features

    • Added a CRD option-source test suite and a package test target to validate annotation-based option sources.
  • Bug Fixes

    • Improved end-to-end install checks with consolidated readiness waits and richer failure diagnostics for releases.
  • Refactor

    • Migrated backup-controller CRD option metadata from legacy vendor extensions to annotation-based keys for cleaner schema metadata.

@coderabbitai

coderabbitai Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: cc344a02-6c5d-43c3-bbd5-64c4e6be496c

📥 Commits

Reviewing files that changed from the base of the PR and between 6039330 and 67b1a96.

📒 Files selected for processing (1)
  • hack/e2e-install-cozystack.bats
🚧 Files skipped from review as they are similar to previous changes (1)
  • hack/e2e-install-cozystack.bats

📝 Walkthrough

Walkthrough

Moves CozyStack CRD dropdown source metadata from per-field x-cozystack-options extensions into CRD-level options.cozystack.io/source.* annotations; updates Go kubebuilder markers, regenerated CRD YAMLs, removes AWK post-processing in codegen, adds CRD validation tests, and refines e2e HelmRelease readiness checks.

Changes

CRD source metadata annotation migration

Layer / File(s) Summary
Go API contract: kubebuilder source metadata annotations
api/backups/v1alpha1/backup_types.go, api/backups/v1alpha1/backupjob_types.go, api/backups/v1alpha1/plan_types.go, api/backups/v1alpha1/restorejob_types.go
Kubebuilder metadata annotations are added to Backup, BackupJob, Plan, and RestoreJob CRD root types, declaring options.cozystack.io/source.* mappings for dropdown reference fields.
Generated CRD YAML schema migration
packages/system/backup-controller/definitions/backups.cozystack.io_*.yaml
Backup, BackupJob, Plan, and RestoreJob CRD YAMLs move source metadata from x-cozystack-options schema extensions into CRD metadata.annotations, removing the per-field vendor extensions while preserving schema types and validations.
Codegen script simplification
hack/update-codegen.sh
Removes the AWK-based inject_cozystack_options post-processing and its invocations; CRD source options are now handled via Go markers and annotations.
CRD contract validation tests
packages/system/backup-controller/Makefile, packages/system/backup-controller/tests/crds-option-sources_test.yaml
Adds Helm unit tests that assert presence of options.cozystack.io/source.* annotations and absence of x-cozystack-options in OpenAPI schemas; Makefile test target runs helm unittest.
E2E test readiness verification improvements
hack/e2e-install-cozystack.bats
Consolidates HelmRelease readiness waits into a single guarded kubectl wait followed by a secondary gated wait; on failure prints detailed per-HelmRelease condition diagnostics before exiting nonzero.

Sequence Diagram(s)

sequenceDiagram
  participant Dev as Developer (add markers)
  participant GoTypes as Go API types
  participant ControllerGen as controller-gen
  participant CRDYAML as Generated CRD YAMLs
  participant Tests as Helm unit tests

  Dev->>GoTypes: add kubebuilder:metadata:annotations markers
  GoTypes->>ControllerGen: controller-gen reads markers
  ControllerGen->>CRDYAML: emit metadata.annotations (options.cozystack.io/source.*)
  Tests->>CRDYAML: validate annotations present and x-cozystack-options absent
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related issues

Suggested labels

size/M, area/platform

Suggested reviewers

  • sircthulhu
  • lllamnyp
  • IvanHunters

Poem

🐰 I nudged the AWK out of sight,
Markers whisper metadata bright.
YAMLs now carry source with care,
Tests hop in to check and share.
A tidy hop — the CRDs flare.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main change: moving dropdown option sources from CRD schema to CRD annotations. It accurately reflects the primary fix in the changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/backup-controller-crd-options

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions github-actions Bot added area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files labels Jun 4, 2026
@lexfrei Aleksei Sviridkin (lexfrei) added the backport Should change be backported on previous release label Jun 4, 2026
…chema

The four backups.cozystack.io CRDs injected the x-cozystack-options vendor
extension into their OpenAPI validation schema. apiextensions JSONSchemaProps
is a closed struct that only preserves x-kubernetes-* vendor extensions, so
helm-controller's server-side apply rejected the unknown key ("field not
declared in schema") and the CRDs never applied — the controller crash-looped
and backups were non-functional.

Move the field-to-source mapping the dashboard needs into CRD
metadata.annotations (preserved by the apiserver), emitted via
+kubebuilder:metadata:annotations markers, and drop the post-processing awk
injector. The dashboard reattaches the extension client-side.

A helm-unittest suite pins the contract: each dropdown field carries its source
in metadata.annotations and must not carry x-cozystack-options in the schema,
guarding against re-introducing the injector.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the fix/backup-controller-crd-options branch from f0e959d to 2363f2d Compare June 4, 2026 09:45
@github-actions github-actions Bot added size/L This PR changes 100-499 lines, ignoring generated files and removed size/M This PR changes 30-99 lines, ignoring generated files labels Jun 4, 2026
@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the fix/backup-controller-crd-options branch from 82ed4e2 to 6901715 Compare June 4, 2026 10:17
@lexfrei
Aleksei Sviridkin (lexfrei) marked this pull request as ready for review June 4, 2026 10:37
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical issue where the cozy-backup-controller failed to reconcile due to invalid vendor extensions in CRD schemas. By moving these mappings to metadata annotations and cleaning up the injection process, the CRDs are now correctly accepted by the apiserver. Additionally, the PR strengthens the CI pipeline by improving how HelmRelease readiness is verified, ensuring that deployment failures are caught and reported clearly.

Highlights

  • CRD Schema Fix: Removed the unsupported 'x-cozystack-options' vendor extension from CRD OpenAPI schemas, which was causing server-side apply failures.
  • Annotation-based Mapping: Migrated field-to-source mappings to CRD metadata annotations using kubebuilder markers, ensuring compatibility with the apiserver.
  • CI/CD Improvements: Enhanced E2E installation tests to properly gate on HelmRelease readiness and provide detailed failure diagnostics.
  • Test Coverage: Added a helm-unittest suite to enforce the new annotation contract and prevent future regressions in CRD schema definitions.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@dosubot dosubot Bot added area/api Issues or PRs related to the cozystack-api aggregated API server kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels Jun 4, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors how dropdown option sources are exposed in the backup-controller CRDs. Instead of injecting the custom x-cozystack-options vendor extension directly into the OpenAPI validation schema (which is rejected by server-side apply), the sources are now defined via metadata.annotations on the CRDs. The awk-based injection script (hack/inject-cozystack-options.awk) has been removed, and the annotations are now generated directly from kubebuilder markers in the Go API types. Additionally, a Helm unittest suite has been added to verify this contract, and the E2E installation script (hack/e2e-install-cozystack.bats) has been updated to improve HelmRelease readiness checks. Feedback on the E2E script highlights potential vulnerabilities to transient kubectl failures and false positives in the readiness check loop, suggesting a more robust check using custom columns.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread hack/e2e-install-cozystack.bats Outdated
Comment on lines +130 to +138
if ! timeout 120 sh -ec 'while kubectl get hr -A --no-headers | grep -v " True " | grep -q .; do sleep 5; done'; then
kubectl get hr -A
# kubectl's STATUS column truncates long messages; dump the full Ready
# condition per non-ready HR so the real error (e.g. a rejected CRD) is
# visible in the test output instead of only inside the cozyreport.
kubectl get hr -A --no-headers | grep -v " True " | while read -r ns name rest; do
echo "--- Non-ready HelmRelease: $ns/$name" >&2
kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2
done

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There are two critical issues with the current readiness check implementation:

  1. Transient kubectl Failure Vulnerability: The loop condition while kubectl get hr -A --no-headers | grep -v " True " | grep -q .; do sleep 5; done will immediately terminate and succeed if kubectl fails transiently (e.g., due to rate limits or temporary API server unavailability). This is because a failure in kubectl results in empty output, which causes grep -q . to exit with 1 (false), ending the loop and letting the test pass silently.
  2. False Positives from Status Messages: Grepping the entire line for " True " can lead to false positives if a non-ready HelmRelease has a status message or description containing the substring " True " (e.g., "reconciliation set to True" or "dependency True is not ready").

Solution

  • Store the kubectl output in a variable and only evaluate readiness if the command succeeded.
  • Use custom-columns to place the Ready condition status as the first column, and then use grep -v "^True" to precisely filter out ready releases, completely avoiding false matches in the namespace, name, or status message.
  if ! timeout 120 sh -ec '
    while true; do
      if out=$(kubectl get hr -A -o custom-columns=READY:.status.conditions[?(@.type=="Ready")].status,NS:.metadata.namespace,NAME:.metadata.name --no-headers 2>/dev/null); then
        if ! printf "%s\n" "$out" | grep -v "^True" | grep -q .; then
          exit 0
        fi
      fi
      sleep 5
    done
  '; then
    kubectl get hr -A
    # kubectl's STATUS column truncates long messages; dump the full Ready
    # condition per non-ready HR so the real error (e.g. a rejected CRD) is
    # visible in the test output instead of only inside the cozyreport.
    kubectl get hr -A -o custom-columns=READY:.status.conditions[?(@.type=="Ready")].status,NS:.metadata.namespace,NAME:.metadata.name --no-headers | grep -v "^True" | while read -r status ns name; do
      echo "--- Non-ready HelmRelease: $ns/$name (status: $status)" >&2
      kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2
    done

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
packages/system/backup-controller/tests/crds-option-sources_test.yaml (1)

20-82: ⚡ Quick win

Well-structured test assertions validate the annotation contract comprehensively.

The four test cases correctly validate that:

  1. Each CRD carries the required options.cozystack.io/source.* annotations in metadata
  2. No x-cozystack-options vendor extensions remain in the OpenAPI schema

The assertion paths accurately navigate the CRD structure, and the expected annotation keys and values match the actual CRD definitions in the context snippets.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/system/backup-controller/tests/crds-option-sources_test.yaml` around
lines 20 - 82, The test assertions correctly verify presence of metadata
annotations (e.g.
metadata.annotations["options.cozystack.io/source.applicationRef.kind"],
metadata.annotations["options.cozystack.io/source.planRef.name"],
metadata.annotations["options.cozystack.io/source.backupClassName"],
metadata.annotations["options.cozystack.io/source.backupRef.name"],
metadata.annotations["options.cozystack.io/source.targetApplicationRef.kind"])
and absence of vendor extensions in the OpenAPI schema (e.g.
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.applicationRef.properties.kind["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.planRef.properties.name["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.backupClassName["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.backupRef.properties.name["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.targetApplicationRef.properties.kind["x-cozystack-options"]);
no code changes are required—keep the assertions as-is, or if you want extra
safety, add one additional assertion per CRD to check the annotation keys exist
before validating values (use the same metadata.annotations[...] paths) to make
failures clearer.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/e2e-install-cozystack.bats`:
- Line 130: The pipeline in the while condition (the command starting with
"kubectl get hr -A --no-headers | grep -v \" True \" | grep -q .") can hide
kubectl failures under POSIX sh because pipeline exit status isn't propagated;
change the loop so you capture kubectl's exit and output explicitly and only
consider the grep result if kubectl succeeded. Concretely, replace the one-liner
with a loop that runs "kubectl get hr -A --no-headers" into a temporary buffer
(or variable), check kubectl's exit status, and then run "grep -v ' True '" and
"grep -q ." against that buffer (sleeping and retrying on either a non-empty
grep result or a transient kubectl failure) so the timeout branch triggers
correctly on kubectl errors.

---

Nitpick comments:
In `@packages/system/backup-controller/tests/crds-option-sources_test.yaml`:
- Around line 20-82: The test assertions correctly verify presence of metadata
annotations (e.g.
metadata.annotations["options.cozystack.io/source.applicationRef.kind"],
metadata.annotations["options.cozystack.io/source.planRef.name"],
metadata.annotations["options.cozystack.io/source.backupClassName"],
metadata.annotations["options.cozystack.io/source.backupRef.name"],
metadata.annotations["options.cozystack.io/source.targetApplicationRef.kind"])
and absence of vendor extensions in the OpenAPI schema (e.g.
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.applicationRef.properties.kind["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.planRef.properties.name["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.backupClassName["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.backupRef.properties.name["x-cozystack-options"],
spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.targetApplicationRef.properties.kind["x-cozystack-options"]);
no code changes are required—keep the assertions as-is, or if you want extra
safety, add one additional assertion per CRD to check the annotation keys exist
before validating values (use the same metadata.annotations[...] paths) to make
failures clearer.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: da1b84a6-3b57-4674-8fb2-6c608f6bdc4c

📥 Commits

Reviewing files that changed from the base of the PR and between adb9989 and 6901715.

📒 Files selected for processing (13)
  • api/backups/v1alpha1/backup_types.go
  • api/backups/v1alpha1/backupjob_types.go
  • api/backups/v1alpha1/plan_types.go
  • api/backups/v1alpha1/restorejob_types.go
  • hack/e2e-install-cozystack.bats
  • hack/inject-cozystack-options.awk
  • hack/update-codegen.sh
  • packages/system/backup-controller/Makefile
  • packages/system/backup-controller/definitions/backups.cozystack.io_backupjobs.yaml
  • packages/system/backup-controller/definitions/backups.cozystack.io_backups.yaml
  • packages/system/backup-controller/definitions/backups.cozystack.io_plans.yaml
  • packages/system/backup-controller/definitions/backups.cozystack.io_restorejobs.yaml
  • packages/system/backup-controller/tests/crds-option-sources_test.yaml
💤 Files with no reviewable changes (2)
  • hack/inject-cozystack-options.awk
  • hack/update-codegen.sh

Comment thread hack/e2e-install-cozystack.bats Outdated

@kvaps Andrei Kvapil (kvaps) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the mapping parity against the old awk targets (all four CRDs, field→source pairs match 1:1), the annotation values against the provider names in pkg/registry/core/option/providers.go, and that the codegen-drift check proves the markers reproduce the committed YAML. The unittest suite pinning the contract in both directions is a nice touch.

Two notes on the e2e gate:

  1. The authoritative re-list can pass vacuously: in kubectl get hr -A --no-headers | grep -v " True " | grep -q ., a failed kubectl call (or empty output) makes the pipeline return 1, the loop exits, and the gate passes — POSIX sh -ec doesn't catch pipeline-internal failures. Worth asserting that kubectl succeeded and the HR list is non-empty (it's guaranteed >0 at this point).
  2. Pre-existing nit: grep -v " True " matches anywhere in the line, so an HR whose STATUS message contains " True " would be hidden; awk '$4 != "True"' on the READY column would be exact.

Neither is blocking. Since the fix is precisely "HR fails to apply", a full e2e run on this PR before merge would be the real proof — and it exercises the restored gate too.

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took another pass at the restored e2e gate. Both bots correctly spotted that the readiness loop can still pass green on a kubectl error, but their fixes keep the hand-rolled loop. Proposing the kubectl wait idiom this file already uses everywhere else instead — details inline.

Comment thread hack/e2e-install-cozystack.bats Outdated
Comment on lines 127 to 141
# Fail the test if any HelmRelease is not Ready. Re-list rather than trust
# the wait above so late-created HRs are gated too; the brief retry absorbs
# momentary Unknown flaps from helm-controller drift reconciles.
if ! timeout 120 sh -ec 'while kubectl get hr -A --no-headers | grep -v " True " | grep -q .; do sleep 5; done'; then
kubectl get hr -A
# kubectl's STATUS column truncates long messages; dump the full Ready
# condition per non-ready HR so the real error (e.g. a rejected CRD) is
# visible in the test output instead of only inside the cozyreport.
kubectl get hr -A --no-headers | grep -v " True " | while read -r ns name rest; do
echo "--- Non-ready HelmRelease: $ns/$name" >&2
kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2
done
echo "Some HelmReleases failed to reconcile" >&2
exit 1
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both bots flagged the hole here and they're right, but their fixes keep the hand-rolled loop. No shell option can rescue this shape:

  • a kubectl/API error → empty stdout → grep -q . exits 1 → the while condition is false → the loop ends and sh exits 0 → the gate passes green. The exact masking class this PR set out to fix.
  • pipefail can't help even where available (sh is dash on the runners): kubectl's error exit is 1 and grep -q's no-match exit is 1 — indistinguishable. The verdict must not come out of a pipeline at all.
  • grep " True " also substring-matches the free-text STATUS message, so a non-ready HR whose message contains True reads as ready.

Everywhere else in this file the readiness gate is kubectl wait ... --for=condition=ready (L153, L201, L277, …). The same idiom works here and makes every failure mode a non-zero exit: non-ready HR (named in the output), API error, even zero HRs matched (no matching resources found). Unknown flaps are absorbed natively — wait waits for the condition to become true within the window, which is what the 120s retry loop was hand-rolling.

Suggested change
# Fail the test if any HelmRelease is not Ready. Re-list rather than trust
# the wait above so late-created HRs are gated too; the brief retry absorbs
# momentary Unknown flaps from helm-controller drift reconciles.
if ! timeout 120 sh -ec 'while kubectl get hr -A --no-headers | grep -v " True " | grep -q .; do sleep 5; done'; then
kubectl get hr -A
# kubectl's STATUS column truncates long messages; dump the full Ready
# condition per non-ready HR so the real error (e.g. a rejected CRD) is
# visible in the test output instead of only inside the cozyreport.
kubectl get hr -A --no-headers | grep -v " True " | while read -r ns name rest; do
echo "--- Non-ready HelmRelease: $ns/$name" >&2
kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2
done
echo "Some HelmReleases failed to reconcile" >&2
exit 1
fi
# Fail the test if any HelmRelease is not Ready. Wait again on a fresh
# listing so HelmReleases created after the snapshot above are gated too;
# the window absorbs momentary Unknown flaps from drift reconciles.
if ! kubectl wait hr --all -A --timeout=2m --for=condition=ready; then
kubectl get hr -A || true
# kubectl's STATUS column truncates long messages; dump the full Ready
# condition per non-ready HR so the real error (e.g. a rejected CRD) is
# visible in the test output instead of only inside the cozyreport.
kubectl get hr -A --no-headers | awk '$4 != "True"' | while read -r ns name _; do
echo "--- Non-ready HelmRelease: $ns/$name" >&2
kubectl get hr -n "$ns" "$name" -o jsonpath='{range .status.conditions[*]}{.type}={.status} reason={.reason}: {.message}{"\n"}{end}' >&2 || true
done
echo "Some HelmReleases failed to reconcile" >&2
exit 1
fi

Notes:

  • awk '$4 != "True"' keys on the READY column positionally instead of substring-matching the whole line.
  • the || trues live only in the failure path: bats aborts on any command error, so the diagnostics stay best-effort while the exit 1 verdict is unconditional.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied verbatim and amended into the e2e commit — the repo merges without squash, so I kept the history at two commits rather than stacking a fixup. Agreed the verdict must not come out of a pipeline: routing it through kubectl wait's own exit code makes a non-ready HR, an API error, and zero matches all non-zero — exactly the masking class this change set out to close. This also folds in the other readiness-check notes on the thread (substring match on the STATUS column, now awk '$4 != "True"' on READY).

Comment thread hack/e2e-install-cozystack.bats Outdated
Comment on lines +112 to +117
# Pacing only (|| true): a backgrounded fan-out's exit codes are discarded
# by a bare POSIX `wait`, which let permanently-failing HRs ship green
# (#2822). A single kubectl wait names every HR that timed out in the
# trace; the authoritative pass/fail gate is the outcome-based re-list
# below, which also covers HRs created after this snapshot.
kubectl wait hr --all -A --timeout=15m --for=condition=ready || true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the gate below becoming a plain kubectl wait, this comment can shrink — the fan-out history lives in #2822.

Suggested change
# Pacing only (|| true): a backgrounded fan-out's exit codes are discarded
# by a bare POSIX `wait`, which let permanently-failing HRs ship green
# (#2822). A single kubectl wait names every HR that timed out in the
# trace; the authoritative pass/fail gate is the outcome-based re-list
# below, which also covers HRs created after this snapshot.
kubectl wait hr --all -A --timeout=15m --for=condition=ready || true
# Pacing only: names every HR that timed out in the trace; the authoritative
# gate re-lists below, covering HRs created after this snapshot (#2822).
kubectl wait hr --all -A --timeout=15m --for=condition=ready || true

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied.

The HR readiness gate in e2e-install-cozystack.bats was toothless in two
layers: the backgrounded kubectl-wait fan-out discarded child exit codes
(POSIX `wait` without args returns 0), and the final check echoed
"Some HelmReleases failed to reconcile" without exiting non-zero — a
regression from 1f24038, which replaced the unavailable bats `fail`
helper with a bare echo. A permanently-failing platform HelmRelease
(e.g. the backup-controller CRD rejection, #2822) shipped through green
CI for weeks.

Replace the fan-out with a single `kubectl wait hr --all -A` for trace
visibility, gate on an outcome-based re-list (which also covers HRs
created after the snapshot, with a 120s retry to absorb momentary
drift-reconcile flaps), and dump the full Ready condition message per
non-ready HR so the real error is visible in the test output.

Refs #2822

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the fix/backup-controller-crd-options branch from 6039330 to 67b1a96 Compare June 4, 2026 12:19

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The e2e gate now uses the file's kubectl wait idiom — every failure mode (non-ready HR, API error, zero HRs matched) exits non-zero, so a permanently-failing HelmRelease can't ship green again. CRD annotation contract is pinned by the helm-unittest suite, and the option sources all resolve against the providers in pkg/registry/core/option/providers.go. LGTM.

@lexfrei
Aleksei Sviridkin (lexfrei) merged commit 43fc969 into main Jun 4, 2026
10 of 11 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the fix/backup-controller-crd-options branch June 4, 2026 12:31
@github-actions

github-actions Bot commented Jun 4, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/api Issues or PRs related to the cozystack-api aggregated API server area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) backport Should change be backported on previous release kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

backup-controller CRDs fail to apply: x-cozystack-options is not valid in apiextensions.k8s.io/v1 schemas

3 participants