Skip to content

[Backport release-1.5] fix(kube-ovn): reload kubeovn-webhook serving certificate on cert-manager renewal - #3879

Merged
myasnikovdaniil merged 6 commits into
release-1.5from
backport-3557-to-release-1.5
Aug 19, 2026
Merged

[Backport release-1.5] fix(kube-ovn): reload kubeovn-webhook serving certificate on cert-manager renewal#3879
myasnikovdaniil merged 6 commits into
release-1.5from
backport-3557-to-release-1.5

Conversation

@github-actions

Copy link
Copy Markdown

Description

Backport of #3557 to release-1.5.

The webhook loaded its TLS key pair once at startup and served it as a
static tls.Config.Certificates entry for the lifetime of the process. A
cert-manager renewal of the backing Secret was never picked up by a
running pod, so once the certificate expired (about a year after install)
the pod kept presenting the expired certificate. The kube-apiserver's TLS
call to the webhook then failed and, because the MutatingWebhookConfiguration
uses failurePolicy: Fail, every pod creation in tenant namespaces was
rejected (including virt-launcher pods, blocking all VMIs).

Serve the certificate through a reloading tls.Config.GetCertificate
callback that re-reads the key pair when the mounted files change and
keeps serving the last good certificate on a failed reload. Add a
regression test that fails against the static implementation and passes
once the certificate is reloaded.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 44d9f53)
renewBefore: 24h on a one-year certificate leaves a one-day window for
cert-manager to rotate the Secret and for the webhook to pick it up.
Widen it to 720h (30 days) so renewal happens well ahead of expiry.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 791d99d)
Address review of the reload path:
- Observe the cert file mtime before reading the key pair, so a Secret
  swap racing the read is retried on the next handshake instead of being
  cached under a newer mtime and missed.
- Record the load attempt regardless of outcome, so a persistently
  unreadable file is retried only once its mtime advances, not on every
  handshake.
- Detect change with mtime inequality instead of a strictly-newer
  comparison, catching equal-mtime replacements and backward clock steps.
- Log stat failures instead of swallowing them, so a broken mount is not
  invisible until expiry.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit a537bb6)
A previous revision recorded the certificate file mtime even when the
reload failed, to avoid re-reading a persistently bad file on every
handshake. That conflated two failure modes: a transient, content-
independent error (fd exhaustion, a torn read) on a VALID renewed file
advanced the recorded mtime, so the staleness check never fired again
and the renewed certificate was never loaded. The cached certificate
then expired within renewBefore, and failurePolicy: Fail blocked all
tenant pod creation - exactly the outage this reloader prevents.

Advance the recorded mtime only on a successful load, and bound retries
of a failed load with a wall-clock interval instead. A transient failure
is now retried until it succeeds, while a persistently unreadable file
is still retried only once per interval, not on every handshake. Keep a
single reload in flight to avoid a thundering herd, and rate-limit the
failure log. Add a regression test that fails when the mtime is advanced
on failure.

Detect change with mtime inequality, so a backward clock step is also
picked up.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 5b6bb47)
…osure

Collapse the certReloader (mtime tracking, serialized refresh, retry backoff,
rate-limited logging, last-good-certificate fallback) into a GetCertificate
callback that simply re-reads the key pair from disk on every handshake. The
signature is unchanged, so main.go is untouched.

The extra machinery guarded scenarios this deployment cannot produce:

- Torn reads are unreachable: the cert/key files come from a plain Secret
  volume with no subPath, and Kubernetes' atomic writer swaps the whole ..data
  directory via a single symlink flip, so a reader always sees a complete old
  or complete new generation of both files.
- The per-handshake LoadX509KeyPair cost is negligible next to the asymmetric
  crypto the handshake already performs, so no cache is needed.
- The webhook configures no mTLS, so GetCertificate not refreshing client CA
  pools does not apply.

If the mounted files genuinely become unreadable (an operator error, not a
renewal) the handshake now fails loudly instead of silently serving a stale
certificate. The core regression test is retained and remains non-vacuous: it
fails against a static tls.Config.Certificates and passes with the reloading
callback.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 8baa449)
The webhook http.Server set no read, write, or idle timeout, so a client that
opens a connection and sends headers slowly could hold a goroutine and file
descriptor indefinitely. Add ReadHeaderTimeout, ReadTimeout, WriteTimeout, and
IdleTimeout.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 4151885)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants