feat(opensearch): add TLS support for HTTP API and Dashboards - #2682

The trust anchor reaches the tenant through two different labels on two
different objects: publish-ca-cert marks the key-bearing source the extraction
controller reads, and tenant-ca marks the key-free projection it writes, which
is what the ApplicationDefinition selects. Nothing enforced that separation.
Because selection is purely by label, renaming one key on the CA Certificate's
secretTemplate would have made the CA private key tenant-readable with every
test in the repo still green — a one-line edit past a boundary the design
calls deliberate.

Pin it from both sides: the CA Certificate must not carry the projection's
label, and the ApplicationDefinition must not select by the source's.

Also give _namespace a default before dereferencing it. The existing check
claimed to fail clearly when the tenant host is missing, but only covered
_namespace present and empty; with the key absent entirely the render died on
a raw nil pointer before the message could be emitted.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

Three corrections, all where the previous wording would have misled someone
making a security decision.

The external access note claimed the published DNS names are the names the
certificate carries SANs for. That is true only under chart-managed TLS. With
TLS operator-managed — turned off explicitly, or fallen back to below
OpenSearch 2.0.0 — the operator issues for in-cluster names only, so a client
reaching the published name cannot verify it at all. Say which mode covers
which names.

Turning chart-managed TLS off does not withdraw the trust anchor. Helm removes
the certificates but cert-manager keeps their Secrets, so the CA keeps
matching the publish label and the tenant keeps receiving it, while the
endpoint has already moved back to an operator-issued certificate under a
different CA. Nothing the chart renders can prevent that — it does not own the
Secret — so name the manual cleanup instead of leaving it to be discovered.

The nil guard on tls was commented as handling an explicit null. Helm coalesces
that back to the default in values.yaml, so nil never reaches the template and
no test can exercise it; the comment now says it is defensive only.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

… bad usage

Enabling chart-managed TLS replaces the certificate the operator was issuing,
so any name the operator covered and the chart does not stops verifying the
moment TLS goes on. The leaf was missing three: <release>.<namespace>,
<release>-dashboards.<namespace>, and the <release>-discovery service. The
two-label form is the one that matters most — with ndots:5 and the default
search list it is what a client in another namespace actually resolves, and
therefore what its TLS verification runs against. This was reachable without
anyone opting in: with external access on and the switch unset, the tri-state
turns TLS on by itself, so an upgrade alone would have narrowed the coverage
of a working release.

Also drop key encipherment from both certificates. RFC 5480 section 3 forbids
it for ECDSA keys, which perform no key transport, and both keys here are
ECDSA P-256 — it was left over from when they were RSA.

The tenant-exposure check now asserts the shape of secrets.include rather than
grepping for one spelling of a name. It is the only thing standing between the
CA private key and the tenant, and the grep passed a leak written any other
way; every entry must now be either the credentials Secret or the key-free
tenant-ca projection.

Correct the external-dns note while here: external-dns runs upsert-only, so
re-applying publishes the new name but never withdraws the old record.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

… needed

Two things a user has to know that the tri-state documentation did not say.

An existing external release has the switch unset and runs operator-managed
TLS today, so it moves to chart-managed TLS the first time it reconciles after
this ships. That rolls the cluster and re-anchors it on a new CA — anything
pinned to the operator's CA has to be repointed. The choice is deliberate and
matches what a new release gets, but it must not be discovered by watching a
data cluster restart, so say it, and pin the resolution in a test of its own
rather than leaving it to incidental coverage.

The manual restart was framed as something rotation needs. It is wider than
that: the certificate is mounted by subPath and never refreshed in place, so
changing external access, the tenant host or the cluster domain also reissues
it while the pods keep serving the old one, with no error to notice. Say that
under both parameters, since external is the one a user actually touches.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

The check that no key-bearing Secret reaches the tenant iterated over whatever
the selector returned, and a loop over nothing is a green test. Restructure
spec.secrets.include, or run where yq is not installed, and it reported ok
while checking nothing at all — on the one assertion standing between the CA
private key and the tenant. Require yq up front, and assert the entry count
before judging the entries, so the list being what we think it is becomes part
of the claim rather than an assumption underneath it.

Also correct what the release-name guard says it protects. Certificate, Issuer
and Secret names are DNS subdomains bounded at 253, which nothing here comes
near; the real limits are the 63-char DNS label on the SANs, where
<release>-dashboards binds at 52, and the 64-char X.509 common name, which
binds at 56. The 47-char guard sits under all of them and is unchanged — only
the reasoning was wrong, and someone relaxing it later would have started from
a premise that is not true. Pin both sides of the boundary now, not just the
rejected one.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

grep exits non-zero when it counts nothing, so under set -e the count assertion
killed the test before its own message could say what was expected. The check
still failed, which is what matters, but it failed as a bare grep trace instead
of naming the invariant it was defending.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…oint

One combined leaf covered both services, so the Dashboards pod was mounted a
private key whose SANs included every OpenSearch API name — the in-cluster
ladder and, with external access on, the public hostname. Dashboards is the
more exposed of the two and the more likely to be compromised, and anything
that took it could then present a valid certificate for the API endpoint. The
ten-year lifetime the subPath mount forces makes that a ten-year window.

The operator never did this: it issues <cluster>-http-cert and
<cluster>-dashboards-cert separately, so collapsing them was a widening this
chart introduced. Issue one certificate per service instead, both from the
same CA so a client trusting the published anchor still verifies both, each
carrying only its own names. The Dashboards certificate renders last and only
when Dashboards is enabled, so the other four keep their positions.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

Splitting the certificate left four statements describing the design it
replaced, one of them with an operational consequence.

The manual procedures named only the API certificate. Changing external access,
the tenant host or the cluster domain reissues both, so an operator following
them literally restored the API endpoint and left Dashboards serving a
certificate that no longer matched its own name — the silent failure the
paragraph exists to prevent. It now covers both sides, and says the Dashboards
deployment restarts separately, since it is not an OpenSearch pod. The
disable-path cleanup likewise left the Dashboards Secret behind.

The template header still described one leaf covering both services, directly
contradicting the explanation further down the same file, and the external
access note still spoke of a single certificate carrying both names.

The Dashboards lifetime comment justified itself by the subPath mount, which is
the one thing that does not apply there: the operator mounts a provided
Dashboards certificate as a whole volume, so the kubelet does refresh it. The
ten years are right for a different reason — Dashboards reads the files once at
startup and nothing restarts it when they change.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…ing has

The API leaf is pinned three ways: it may not take a name the operator
generates into, it may not be offered to the CA-extraction controller as a CA
source, and it may not carry the other service's names. The dashboards
certificate inherits every one of those hazards and had only positive
assertions, so all three failures passed the suite green — including undoing
the split itself by putting the API names back on the key the more exposed pod
holds.

Mirror the three, and the ECDSA usages assertion with them, so each fails
against the corresponding regression instead of documenting the safe shape and
checking none of it.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…ly what it must not

The guards added for the dashboards certificate were all prohibitive, so it
ended up well defended against being the wrong thing and unchecked on being
the right one. Pointing its issuerRef at the bootstrap self-signed Issuer
instead of the shared CA passed the whole suite: cert-manager still issues,
Dashboards still serves HTTPS, the release still goes Ready, and the anchor
the platform publishes silently stops verifying that endpoint. So did
switching the key to RSA, reintroducing the short lifetime that expires in
place, and pointing it at the API certificate's Secret so two Certificates
race over one object while the CR names only one of them.

Mirror the structural assertion the API leaf already has — kind, secret name,
issuer, key algorithm, size and rotation policy — and extend the lifetime pin.
That also establishes that the document really is the dashboards certificate,
without which the prohibitive guards would pass against whatever landed at
that position if the order ever moved.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…tificate

Neither leaf pinned isCA, and no other assertion can stand in for it. certSign
is derived from that field rather than from spec.usages, so flipping a leaf to
isCA: true renders an unchanged usages list, satisfies the exact-length and
no-certSign pins, and still issues a working intermediate CA — one whose
private key lives in the pod and chains to the anchor tenants are handed. That
is a wider hole than the key-sharing the per-service split closes: it signs for
any name, not just the sibling service's.

issuerRef was pinned by name but not by kind, and on the dashboards leaf only.
A reference that resolves to a different object of the same name still issues,
still serves, and still goes Ready while anchoring on something the published
trust anchor does not cover — the same silent failure already argued for the
dashboards leaf, which applies just as well to the more exposed API endpoint
and to the CA itself. Pin kind and group everywhere the name is pinned.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

The comparison was described as running against the resolved image tag. It runs
against the resolved version string — the same value that goes into
spec.general.version, which is the field the operator branches on. That
distinction is the reason the floor is correct: an images.opensearch override
changes the running image without moving spec.general.version, so it cannot
put the operator on one side of 2.0.0 while the chart believes the other.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

The external Dashboards Service selected opster.io/opensearch-cluster together
with app.kubernetes.io/component. The operator labels Dashboards pods with
opensearch.cluster.dashboards and nothing else — app.kubernetes.io appears
nowhere in it — so the selector matched no pod and the LoadBalancer stood up
with no endpoints and no error to show for it.

That was survivable while nothing pointed at it. It is not now: the certificate
carries a SAN for the Dashboards hostname and the parameter documentation
describes it as a reachable external endpoint, so the chart promises something
the Service cannot deliver. Select the label that exists, and pin it against
the two that do not.

While here, correct what the SAN comment claims about parity with the operator.
The fully-qualified rung is deliberately different: the operator builds it from
its own DNS_BASE, left at the upstream cluster.local default, while the chart
uses the cluster domain the platform runs on. The chart's name is the one that
resolves — the difference is a fix, not a gap.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

Listing the anchor in the ApplicationDefinition was only half the wiring. That
entry feeds the lineage webhook, which labels the Secret as a tenant resource
for the registry; it is not an access grant. Direct read comes from the chart's
own Role, and that still named only the credentials Secret — so a tenant with
use access got 403 on the very anchor this change exists to hand them. kafka
carries both legs for its CA Secrets; this brings opensearch in line.

Only the key-free projection is granted. The cert-manager CA and both leaves
hold a private key and must never appear in that list, so the new test asserts
the grant is exactly two names and that none of the three key-bearing Secrets
is among them.

The bats case named itself after exposing the CA to the tenant while asserting
only the label; it now says which half it checks and points at the other.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

… empty

Requiring the tenant host turned a supported platform state into a failed
release. A cluster with no root host leaves that value empty all the way down
the tenant tree, and Helm's required rejects the empty string, so external:
true stopped rendering — not just the external-dns annotation but the whole
chart, taking the OpenSearchCluster CR, the credentials Secret and the RBAC
with it. A running release would have gone not-Ready on its next reconcile,
and it fired with chart-managed TLS off too, reaching users who had opted out
of this feature entirely. The same configuration renders on main.

Degrade instead. The LoadBalancers still get addresses and still serve; the
external-dns hostname is simply not published when there is no name to publish,
and the certificate keeps its in-cluster SANs and omits the external ones for
the same reason. Both are asserted, including that the cluster itself still
renders, which is the part whose loss actually hurt.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

The empty-host case rendered with dashboards disabled, so it only ever reached
the API certificate. Breaking the guard on the dashboards leaf alone left the
suite fully green while that certificate emitted a trailing-dot SAN — on the
more externally exposed of the two services.

Enable dashboards in that case and assert the fifth document the same way as
the fourth. Breaking either guard alone now fails.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…uire

The guard rejected any release name over 47 characters. The platform's own cap
is 53, so names between the two are reachable, render today, and stopped
rendering here — including on the auto-on path, where a release that never
asked for chart-managed TLS would have stopped templating entirely and taken
its HelmRelease NotReady.

Nothing was violated at those lengths. The binding constraint is the 63-char
DNS label on the SANs, and which SAN binds depends on configuration: with
Dashboards enabled it is <release>-dashboards at 52, otherwise
<release>-discovery at 53. The old message even printed the evidence, blaming
a dashboards SAN that is not emitted when Dashboards is off.

Compute the limit from the longest label actually built, and name that label in
the message. The only rejected case left is one where the certificate genuinely
could not be issued. Both boundaries are pinned from both sides, and the
long-named auto-on release is pinned as still rendering.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

The bound was derived from certificate SAN labels alone, which missed the
longest name the chart produces: the <release>-dashboards-external Service, at
20 characters of suffix. Service names are DNS-1035 labels capped at 63, unlike
the Certificate and Secret names the reasoning was based on, which are
subdomains capped at 253 and never bind. So the real limit with external access
and Dashboards both on is 43, and relaxing the guard to 52 made a configuration
the API server rejects reachable — the Service is refused and the release fails
with nothing in the chart to explain why.

The bound also has nothing to do with TLS. That Service renders on external
access alone, so a guard living only in the certificate template could not see
the case at all.

Move the check into a helper that picks the longest suffix the current values
actually build — the external Dashboards Service, else the Dashboards name,
else the discovery SAN, else the external Service — and invoke it from both
templates, so it applies with chart-managed TLS off as well. Each boundary is
pinned from both sides, and the message names the object that binds.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

The guard ran from the certificate and external Service templates, so a release
with Dashboards enabled but external access and chart-managed TLS both off
reached no invocation. The operator still creates a <release>-dashboards
Service for it — a DNS-1035 label bounded at 63 — so a 53-character release
rendered clean and then failed at the API server with no chart-level error.
That is the same shape as the external Service gap closed previously, on the
axis that was left behind.

Invoke the guard from the cluster template, which renders unconditionally and
therefore covers every remaining path. The helper is idempotent, so the paths
that already checked are unaffected.

Also drop the limits table left behind in the certificate template. It still
described a 52/53 bound with no mention of the external Service, contradicting
the helper that now owns those numbers — and a stale second copy of that
reasoning is exactly what produced the original mistake.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…rong

The RBAC comment pointed at kafka as carrying both a registry entry and a
read grant. Kafka grants its two CA secrets by name and has no label
selector at all, so a reader following that pointer found a counter-example
rather than the pattern being described.

The name table called -discovery a TLS-only SAN label. The operator creates
that Service on every path, so the name exists with TLS off too. Record
which bounds Helm's own 53-character release-name limit leaves reachable
while the table is being corrected.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

Two suites in the TLS file carried no name, so both reported as a blank
PASS line and a failure in either gave no clue which one broke.

The readme check named for asserting the parameter appears outside the TLS
section only asserted that it appears at all; the section half is the
preceding test. Name it for what it does — it is the guard that stops that
preceding test passing on a README that dropped the row entirely.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…tlives the pin

The comment justified the matched lifetime entirely by the subPath mount
shape, which is a property of the pinned operator rather than of OpenSearch:
upstream has since moved those branches back to whole-directory mounts, so a
routine vendored-chart bump would quietly retire the stated rationale while
the conclusion stayed correct.

State the version-independent reason first — certificates are read once at
startup and no pod restarts on change — and mark the mount-shape half as
pinned-version detail.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>