[Backport release-1.5] fix(kube-ovn): reload kubeovn-webhook serving certificate on cert-manager renewal - #3879
Merged
Conversation
The webhook loaded its TLS key pair once at startup and served it as a static tls.Config.Certificates entry for the lifetime of the process. A cert-manager renewal of the backing Secret was never picked up by a running pod, so once the certificate expired (about a year after install) the pod kept presenting the expired certificate. The kube-apiserver's TLS call to the webhook then failed and, because the MutatingWebhookConfiguration uses failurePolicy: Fail, every pod creation in tenant namespaces was rejected (including virt-launcher pods, blocking all VMIs). Serve the certificate through a reloading tls.Config.GetCertificate callback that re-reads the key pair when the mounted files change and keeps serving the last good certificate on a failed reload. Add a regression test that fails against the static implementation and passes once the certificate is reloaded. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit 44d9f53)
renewBefore: 24h on a one-year certificate leaves a one-day window for cert-manager to rotate the Secret and for the webhook to pick it up. Widen it to 720h (30 days) so renewal happens well ahead of expiry. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit 791d99d)
Address review of the reload path: - Observe the cert file mtime before reading the key pair, so a Secret swap racing the read is retried on the next handshake instead of being cached under a newer mtime and missed. - Record the load attempt regardless of outcome, so a persistently unreadable file is retried only once its mtime advances, not on every handshake. - Detect change with mtime inequality instead of a strictly-newer comparison, catching equal-mtime replacements and backward clock steps. - Log stat failures instead of swallowing them, so a broken mount is not invisible until expiry. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit a537bb6)
A previous revision recorded the certificate file mtime even when the reload failed, to avoid re-reading a persistently bad file on every handshake. That conflated two failure modes: a transient, content- independent error (fd exhaustion, a torn read) on a VALID renewed file advanced the recorded mtime, so the staleness check never fired again and the renewed certificate was never loaded. The cached certificate then expired within renewBefore, and failurePolicy: Fail blocked all tenant pod creation - exactly the outage this reloader prevents. Advance the recorded mtime only on a successful load, and bound retries of a failed load with a wall-clock interval instead. A transient failure is now retried until it succeeds, while a persistently unreadable file is still retried only once per interval, not on every handshake. Keep a single reload in flight to avoid a thundering herd, and rate-limit the failure log. Add a regression test that fails when the mtime is advanced on failure. Detect change with mtime inequality, so a backward clock step is also picked up. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit 5b6bb47)
…osure Collapse the certReloader (mtime tracking, serialized refresh, retry backoff, rate-limited logging, last-good-certificate fallback) into a GetCertificate callback that simply re-reads the key pair from disk on every handshake. The signature is unchanged, so main.go is untouched. The extra machinery guarded scenarios this deployment cannot produce: - Torn reads are unreachable: the cert/key files come from a plain Secret volume with no subPath, and Kubernetes' atomic writer swaps the whole ..data directory via a single symlink flip, so a reader always sees a complete old or complete new generation of both files. - The per-handshake LoadX509KeyPair cost is negligible next to the asymmetric crypto the handshake already performs, so no cache is needed. - The webhook configures no mTLS, so GetCertificate not refreshing client CA pools does not apply. If the mounted files genuinely become unreadable (an operator error, not a renewal) the handshake now fails loudly instead of silently serving a stale certificate. The core regression test is retained and remains non-vacuous: it fails against a static tls.Config.Certificates and passes with the reloading callback. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit 8baa449)
The webhook http.Server set no read, write, or idle timeout, so a client that opens a connection and sends headers slowly could hold a goroutine and file descriptor indefinitely. Add ReadHeaderTimeout, ReadTimeout, WriteTimeout, and IdleTimeout. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit 4151885)
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #3557 to
release-1.5.