[Backport release-1.6] fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy - #3818
Merged
Conversation
A sharded helm-controller inherits HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the flux-aio helm-controller container. A standalone shard needs no external egress (source-controller fetches artifacts), so behind an unreachable corporate proxy it makes a blocking startup HTTPS call that never completes, the manager never serves /healthz, and the liveness probe crashloops the pod forever, freezing every HelmRelease carrying a sharding key. flux-aio itself survives only because it starts once and never restarts. Drop the proxy env (and the now-pointless NO_PROXY) when cloning, alongside the existing localhost/KubePrism sanitisation, so the shard talks to the in-cluster apiserver directly. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit c475faa)
The cloned shard Deployment inherits only a liveness probe (~30s window, no startupProbe), so a controller still syncing caches on startup is killed before it serves /healthz and never recovers. Derive a startupProbe from the liveness handler with a generous budget so a slow start is not fatal, while liveness still catches a wedged running pod. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit d44c87d)
Review follow-ups: - Do not force the cloned startupProbe TimeoutSeconds to 1: keep the value inherited from the liveness handler, otherwise a future flux-aio with a larger liveness timeout would get a stricter startup probe and could recreate the crashloop. Covered by a regression guard (liveness timeout 5s must be inherited). - Soften the proxy-strip rationale to the A/B-confirmed mechanism (blocking external HTTPS call at startup) and note the remote spec.kubeConfig exception that does not apply to cozystack in-cluster guest apiservers. - Correct the test fixture comment: the proxy env is operator/patch-level, not installer-injected (the installer injects only KUBERNETES_SERVICE_*). Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit affdd70)
…sanitisation list The BuildShardDeployment doc comment and the operator README are the only record of how far the cloned shard container has diverged from flux-aio. Add the two steps this fix introduced (dropping the inherited corporate-proxy env and deriving a startupProbe) so the list stays the authoritative account and does not invite re-adding what was deliberately removed. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit 036a31b)
…ct alias regression Model the source liveness probe as flux-aio actually ships it (a bare httpGet with no timing fields) and tighten the startupProbe assertions: - assert the complete normalised budget (delay/period/success/failure) instead of only FailureThreshold, so a later PeriodSeconds regression cannot silently restore a short crashloop window; - assert TimeoutSeconds is inherited from the source probe rather than a hardcoded value, so forcing it stricter is caught; - assert the startupProbe is a DeepCopy of liveness, not an alias, which would otherwise stamp the 30-failure startup budget onto liveness and produce a never-failing liveness probe. The alias assertion is non-vacuous: it fails when the DeepCopy is regressed to a plain alias. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit a10fcb3)
Name all six proxy env spellings in the sanitisation list, and replace the unestablished 'external HTTPS call' / 'flux-aio does not restart' claims with the grounded in-cluster mechanism: after the KUBERNETES_SERVICE_HOST drop above, the shard talks to the management apiserver over the kubelet-injected ClusterIP, which NO_PROXY=.svc does not match, so that startup call is what stalls behind an unreachable proxy. Comment-only; no behaviour change. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> (cherry picked from commit 66df3b2)
Merged
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #3546 to
release-1.6.