Skip to content

feat(platform): ACME wildcard certificate on the default ingress-nginx path - #2988

Merged
Aleksei Sviridkin (lexfrei) merged 2 commits into
mainfrom
feat/ingress-acme-wildcard
Jun 23, 2026
Merged

feat(platform): ACME wildcard certificate on the default ingress-nginx path#2988
Aleksei Sviridkin (lexfrei) merged 2 commits into
mainfrom
feat/ingress-acme-wildcard

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

What this PR does

On the default ingress-nginx path (gateway.enabled=false) every published hostname still minted its own per-host ACME certificate via ingress-shim, even with a DNS-01 solver configured. That hits the Let's Encrypt rate limit (50 certs per registered domain per week) once a deployment exceeds ~50 endpoints. The Gateway API path already issues a single per-apex wildcard in DNS-01 mode; this brings the ingress-nginx path to parity. It implements the remaining ingress-nginx gap that #2400 was narrowed to in its triage comment (DNS-01 is already multi-provider, and the Gateway path already issues wildcards).

The new publishing.certificates.wildcard toggle is opt-in and OFF by default. When set to true with solver=dns01 and no operator-provided wildcard Secret, the platform issues one shared wildcard Certificate for <root-host> + *.<root-host> and serves the resulting Secret as the publishing controller's --default-ssl-certificate. The system service Ingresses (dashboard, grafana, keycloak, harbor, …) then stop requesting a per-host cert.

How it works:

  • core/platform apps.yaml computes the effective wildcard Secret name and a new _cluster.wildcard-issue signal. An operator-provided wildcardSecretName (the BYO path) always wins and never triggers issuance; HTTP-01 cannot issue wildcards; and when Gateway API is enabled the TenantGateway controller issues the wildcard instead — so all three are excluded from auto-issuance.
  • extra/ingress renders the wildcard Certificate only on the publishing controller (Release.Namespace == expose-ingress), because the Secret must be same-namespace for ingress-nginx to read it. The Certificate references the DNS-01 ClusterIssuer that cert-manager-issuers already renders in dns01 mode, so no per-tenant Issuer is minted.

This reuses the existing wildcard-secret consumption path, so no per-service Ingress template changes were needed — only the issuance side is new.

Why opt-in (default off): the chosen wildcard Secret name rides the _cluster channel, which every child tenant inherits verbatim, so this is not root-tenant scoped. A single-label wildcard *.<root-host> covers <service>.<root-host> but not a custom service host outside it (a keycloak ingress.host, harbor host, or grafana host pointed at another domain) nor a child tenant's nested host (<service>.<tenant>.<root-host>). Enabling issuance makes every such service drop its per-host ACME cert and fall back to the default certificate, which does not cover it. Leaving it off by default means a dns01 cluster is never silently switched on upgrade; an operator who enables it accepts the same coverage responsibility as the operator-provided wildcard path, which propagates identically. The tradeoff is pinned by contract tests (keycloak custom host, bucket nested child-tenant host). Per-tenant wildcards remain on the Gateway path and are tracked separately.

Docs: cozystack/website#588

Closes #2400. Part of #2811.

Screenshots

N/A — no UI changes.

Release note

feat(platform): add opt-in shared wildcard certificate issuance on the default ingress-nginx path via publishing.certificates.wildcard (default false). When enabled with a DNS-01 solver, the platform issues one *.<root-host> wildcard for system services instead of a per-host ACME certificate, avoiding Let's Encrypt rate limits at scale.

Summary by CodeRabbit

  • New Features

    • Added automatic wildcard certificate issuance support for DNS-01 ACME solver.
    • New certificates.wildcard configuration option to enable shared wildcard TLS certificates.
    • Ingress rules now properly utilize wildcard certificates when enabled.
  • Tests

    • Added comprehensive test suites validating wildcard certificate generation and ingress integration across multiple scenarios.

@coderabbitai

coderabbitai Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 022912e9-b3bf-46f2-8a2a-36f2fad5dad2

📥 Commits

Reviewing files that changed from the base of the PR and between 82b8c46 and 4218e4c.

📒 Files selected for processing (7)
  • packages/core/platform/templates/apps.yaml
  • packages/core/platform/tests/apps_wildcard_acme_test.yaml
  • packages/core/platform/values.yaml
  • packages/extra/ingress/templates/wildcard-certificate.yaml
  • packages/extra/ingress/tests/wildcard_certificate_test.yaml
  • packages/system/bucket/tests/ingress_wildcard_test.yaml
  • packages/system/keycloak/tests/ingress_wildcard_test.yaml

📝 Walkthrough

Walkthrough

Introduces opt-in DNS-01 wildcard certificate auto-issuance for the default ingress-nginx path. A new certificates.wildcard boolean is added to platform values. The apps.yaml template now conditionally derives wildcard-issue and wildcard-secret-name and propagates them via the cozystack-values Secret. A new cert-manager Certificate Helm template in the ingress package renders the wildcard resource when issuance is enabled. Test coverage is added across platform, ingress, bucket, and keycloak packages.

Changes

DNS-01 Wildcard Certificate Auto-Issuance (WS2)

Layer / File(s) Summary
Platform values contract: certificates.wildcard flag
packages/core/platform/values.yaml
Adds certificates.wildcard: false with inline documentation for DNS-01-only issuance, gateway exclusion, BYO secret precedence, and coverage constraints.
apps.yaml: derive wildcard-issue and wildcard-secret-name
packages/core/platform/templates/apps.yaml
Replaces the prior pass-through block with conditional logic: sets wildcard-issue=true and wildcard-secret-name=cozystack-wildcard-tls when solver=dns01, certificates.wildcard=true, gateway.enabled=false, and no BYO secret; otherwise keeps wildcard-issue=false.
Platform apps.yaml wildcard issuance tests
packages/core/platform/tests/apps_wildcard_acme_test.yaml
Six test cases asserting: default no-issuance (http01), dns01 without opt-in, dns01 with opt-in enabled, gateway suppression, BYO secret precedence, and http01 never issuing wildcards.
ingress package: wildcard cert-manager Certificate template
packages/extra/ingress/templates/wildcard-certificate.yaml
New Helm template that reads _cluster values, validates inputs via fail, and conditionally emits a cert-manager Certificate with DNS-01 wildcard dnsNames on the publishing tenant namespace.
ingress package: wildcard Certificate rendering tests
packages/extra/ingress/tests/wildcard_certificate_test.yaml
Full test suite covering successful rendering, non-rendering when disabled or off-tenant, issuer-name override, and fail-fast errors for missing wildcard-secret-name and root-host.
Downstream ingress wildcard mode tests (bucket, keycloak)
packages/system/bucket/tests/ingress_wildcard_test.yaml, packages/system/keycloak/tests/ingress_wildcard_test.yaml
Bucket adds a nested child-tenant scenario asserting per-host ACME is dropped in wildcard mode; keycloak adds two dns01 custom-host scenarios verifying annotation and TLS secretName presence/absence based on wildcard-secret-name.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

  • #2400 — This PR directly implements the proposed wildcard certificate mode: it introduces a certificates.wildcard opt-in, DNS-01-driven Certificate resources, and ingress-level suppression of per-host ACME in favor of the shared wildcard secret, addressing the Let's Encrypt rate-limit bottleneck described in the issue.
  • Unified TLS certificate and external-exposure model #2811 — This PR implements WS2 of the unified TLS model epic by introducing the conditional wildcard-issuance logic to the ingress-nginx default path, directly advancing that workstream's stated goal.
  • feat(platform): propagate operator wildcard certificate to per-tenant ingress controllers #2820 — This PR propagates wildcard-secret-name and wildcard-issue through cozystack-values to child-tenant ingress controllers, which directly enables the goal described in that issue.

Possibly related PRs

  • cozystack/cozystack#2819: Both PRs wire _cluster.wildcard-secret-name through Helm rendering into wildcard/Gateway certificate handling, with this PR covering the ingress-nginx issuance path and the referenced PR covering the TenantGateway existingSecret path.

Suggested labels

area/networking

Suggested reviewers

  • kvaps
  • lllamnyp
  • androndo
  • IvanHunters

🐇 Hoppity-hop, the certs come in one,
No more per-host ACME — that race has been run!
With wildcard: true and dns01 in play,
A single *.example.org keeps rate-limits at bay.
The rabbit signs off — wildcard day! 🌟

🚥 Pre-merge checks | ✅ 6
✅ Passed checks (6 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'feat(platform): ACME wildcard certificate on the default ingress-nginx path' clearly and concisely summarizes the main feature being added—opt-in wildcard certificate support for the ingress-nginx path.
Linked Issues check ✅ Passed The PR fully addresses the remaining ingress-nginx wildcard gap identified in #2400, implementing platform-level wildcard certificate support with opt-in toggle, DNS-01 solver integration, and proper Certificate resource generation.
Out of Scope Changes check ✅ Passed All changes are directly scoped to implementing wildcard certificate support for ingress-nginx: platform configuration, conditional logic, Certificate templates, and corresponding test suites validate the feature.
Description check ✅ Passed The PR includes comprehensive documentation explaining the feature's purpose, implementation details, design decisions, limitations, and how it addresses the Let's Encrypt rate limiting issue.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/ingress-acme-wildcard

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) kind/feature Categorizes issue or PR as related to a new feature size/L This PR changes 100-499 lines, ignoring generated files labels Jun 22, 2026
On the default ingress-nginx path (gateway.enabled=false) a DNS-01 solver
still minted a per-host ACME certificate for every system service, hitting
Let's Encrypt rate limits once a deployment exceeds ~50 endpoints. Add an
opt-in publishing.certificates.wildcard toggle (default false) and compute,
in apps.yaml, an effective wildcard secret name plus a wildcard-issue
signal: when enabled with solver=dns01 and no operator-provided
wildcardSecretName, the platform targets one shared wildcard cert
(*.<root-host> + <root-host>) instead of per-host certs.

Issuance is off by default because a single-label wildcard does not cover
a custom service host outside *.<root-host> (e.g. a keycloak ingress.host,
harbor host, or grafana host pointed at another domain); enabling it would
otherwise switch such a host to the controller's default certificate and
serve the wrong certificate. The operator-provided (BYO) name always wins
and never triggers issuance; http01 cannot issue wildcards; and the
Gateway API path issues its own wildcard via the controller, so all three
are excluded. This reuses the existing wildcard-secret-name consumption
path, so no per-service ingress template changes are needed.

Part of #2811.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
When the platform signals shared wildcard issuance via
_cluster.wildcard-issue, the publishing controller (Release.Namespace ==
expose-ingress) renders one cert-manager Certificate for <root-host> +
*.<root-host>. The resulting Secret is served as the controller's default
SSL certificate and the existing wildcard-secret-name consumption path
makes the system services drop per-host ACME. The Certificate uses the
DNS-01 ClusterIssuer that cert-manager-issuers already renders in dns01
mode, so no per-tenant Issuer is minted. Only the publishing controller
renders it — the Secret must be same-namespace for ingress-nginx to read
it.

Part of #2811.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements shared wildcard certificate issuance for the ingress-nginx path to mitigate Let's Encrypt rate limits at scale. By consolidating certificate management for system services into a single wildcard, the platform achieves parity with the Gateway API path. The feature is strictly opt-in to protect existing deployments that may rely on custom service hosts or specific tenant configurations that fall outside the scope of a single-label wildcard.

Highlights

  • Shared Wildcard Issuance: Introduced an opt-in mechanism to issue a single shared wildcard certificate (*.) for system services on the ingress-nginx path when using DNS-01, reducing Let's Encrypt rate limit pressure.
  • Configuration Toggle: Added publishing.certificates.wildcard (default: false) to enable this feature, ensuring backward compatibility and preventing silent upgrades for clusters with custom service hosts.
  • Contract Testing: Added comprehensive contract tests to validate the issuance logic and document the trade-offs regarding host coverage and child-tenant inheritance.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@dosubot dosubot Bot added the area/networking Issues or PRs related to networking (ingress, gateway, vpn, metallb, kube-ovn) label Jun 22, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for shared ACME wildcard certificate issuance on the default ingress-nginx path when using a DNS-01 solver. It adds a new opt-in configuration option publishing.certificates.wildcard in values.yaml, updates the platform templates to compute wildcard settings, and introduces a new wildcard-certificate.yaml template in the ingress package to render the cert-manager Certificate. Comprehensive unit tests have been added across several packages to verify the wildcard issuance logic. The reviewer noted that values.schema.json should be regenerated using make generate to include the new configuration option.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

# its self-signed default cert (browser warning), then self-heals —
# the same delay per-host ACME has on first issue, which an
# operator-provided wildcardSecretName avoids only by pre-existing.
wildcard: false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

A new configuration option publishing.certificates.wildcard has been added to values.yaml, but the corresponding values.schema.json file has not been updated in this pull request.

Please run make generate to regenerate the schema file and ensure the new property is included for validation and dashboard UI integration.

References
  1. The repository style guide states that packages follow an umbrella chart pattern containing a values.schema.json for dashboard UI and input validation, which is regenerated via make generate. (link)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The core/platform chart has no values.schema.json and no make generate target — its publishing.certificates.* keys (solver, issuerName, wildcardSecretName, dns01.*) are documented via values.yaml comments only, not a generated schema. wildcardSecretName was added the same way in #2819. The values.schema.json / make generate convention applies to the catalog charts under packages/apps and packages/extra, not this core chart, so there is nothing to regenerate here. The new wildcard key is documented in values.yaml and in cozystack/website#588.

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — clean, well-guarded opt-in feature; ingress-nginx path now reaches parity with the Gateway wildcard path.

Reviewed for the central safety property (default-off, existing installs unchanged) and the dns01/wildcard wiring. Both hold.

Opt-in / default-off (verified)

  • The issuance gate is a 4-way and: not $byoWildcardsolver==dns01wildcardnot gateway.enabled. Under shipped defaults (wildcard=false, solver=http01, gateway.enabled=false) it never fires.
  • Toggle-off render is byte-additive: wildcard-secret-name stays "", plus one new wildcard-issue: "false" line on the _cluster channel (harmless to existing consumers, which index specific keys). The pre-existing BYO test (apps_wildcard_secret_test.yaml) still passes unchanged.

dns01 / wildcard wiring (verified)

  • The Certificate references the letsencrypt-prod/-stage ClusterIssuer by name, and cert-manager-issuers/cluster-issuers.yaml attaches the multi-provider dns01 solver to exactly that issuer when solver=dns01 — so no per-tenant Issuer is minted and the solver isn't hardcoded to one provider. SAN covers apex + *.<root-host> as intended.
  • Fail-fast guards (fail on wildcard-issue=true with empty secret-name or empty root-host) prevent a Certificate pointing at a nonexistent secret / empty dnsNames if apps.yaml ever regresses.

Consumption / parity (verified)

  • Setting wildcard-secret-name is the single trigger that (a) drops per-host ACME on every system-service Ingress, (b) sets --default-ssl-certificate, (c) renders the one Certificate — services consume the wildcard, they don't double-issue. This is the rate-limit win.
  • Same-namespace default-ssl-certificate ($exposeIngress/$wildcardSecret), gated to the publishing controller (Release.Namespace == expose-ingress); child controllers can't read another ns's secret and are correctly excluded.
  • extra/gateway/tenantgateway.yaml already issues a per-apex wildcard in dns01 mode — the ingress path mirrors it, and the gateway path is explicitly excluded from auto-issuance, so no overlap.

Tests: ran helm unittest on all four touched packages locally — platform 65/65, ingress 15/15, keycloak 4/4, bucket 3/3. The coverage tradeoff (custom host / nested child-tenant host outside *.<root-host>) is pinned as explicit contract tests rather than left implicit. Nice.

Schema: the gemini-bot schema-regen flag is a non-issue — core/platform is a core chart with no values.schema.json and no make generate target (the wildcard key follows the same comment-documented pattern as wildcardSecretName from #2819). Already addressed in-thread.

Not tested against a live cluster — a real DNS-01 wildcard issuance needs a delegated domain + solver, so this is a static + render review. The documented first-issuance browser-warning window and the coverage/blast-radius caveats are honest and match why this is correctly opt-in.

@lexfrei
Aleksei Sviridkin (lexfrei) merged commit 51c5273 into main Jun 23, 2026
14 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the feat/ingress-acme-wildcard branch June 23, 2026 10:13
Aleksei Sviridkin (lexfrei) added a commit to cozystack/website that referenced this pull request Jun 30, 2026
## What this PR does

Document two `publishing.certificates.*` options in the platform-package
value table that were missing from the reference docs.

- `publishing.certificates.wildcard` (new): opt-in shared wildcard
certificate issuance on the default ingress-nginx path. When enabled
with a DNS-01 solver it issues one `*.<root-host>` wildcard for system
services instead of a per-host ACME certificate, avoiding Let's Encrypt
rate limits at scale. Documented with its default (`false`), the dns01 /
gateway-disabled gating, and the coverage / blast-radius caveat.
- `publishing.certificates.wildcardSecretName` (pre-existing, previously
undocumented): operator-provided wildcard TLS Secret that platform
services serve under instead of minting per-host ACME certificates.

Documents the code change in cozystack/cozystack#2988. Part of
cozystack/cozystack#2811.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Documentation**
* Added documentation for new Platform Package Publishing certificate
configuration options enabling wildcard certificate support with opt-in
capability and custom TLS Secret assignment.
* Documented wildcard certificate behavior, including precedence rules
between configuration options, override conditions, and hostname
coverage considerations for tenant and service host patterns.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/networking Issues or PRs related to networking (ingress, gateway, vpn, metallb, kube-ovn) area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) kind/feature Categorizes issue or PR as related to a new feature size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[platform] Add wildcard certificate support for multi-tenant deployments

2 participants