Skip to content

[Backport release-1.6] fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy - #3818

Merged
myasnikovdaniil merged 6 commits into
release-1.6from
backport-3546-to-release-1.6
Aug 18, 2026
Merged

[Backport release-1.6] fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy#3818
myasnikovdaniil merged 6 commits into
release-1.6from
backport-3546-to-release-1.6

Conversation

@github-actions

Copy link
Copy Markdown

Description

Backport of #3546 to release-1.6.

A sharded helm-controller inherits HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the
flux-aio helm-controller container. A standalone shard needs no external
egress (source-controller fetches artifacts), so behind an unreachable
corporate proxy it makes a blocking startup HTTPS call that never completes,
the manager never serves /healthz, and the liveness probe crashloops the pod
forever, freezing every HelmRelease carrying a sharding key. flux-aio itself
survives only because it starts once and never restarts.

Drop the proxy env (and the now-pointless NO_PROXY) when cloning, alongside
the existing localhost/KubePrism sanitisation, so the shard talks to the
in-cluster apiserver directly.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit c475faa)
The cloned shard Deployment inherits only a liveness probe (~30s window,
no startupProbe), so a controller still syncing caches on startup is killed
before it serves /healthz and never recovers. Derive a startupProbe from the
liveness handler with a generous budget so a slow start is not fatal, while
liveness still catches a wedged running pod.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit d44c87d)
Review follow-ups:
- Do not force the cloned startupProbe TimeoutSeconds to 1: keep the value
  inherited from the liveness handler, otherwise a future flux-aio with a
  larger liveness timeout would get a stricter startup probe and could
  recreate the crashloop. Covered by a regression guard (liveness timeout 5s
  must be inherited).
- Soften the proxy-strip rationale to the A/B-confirmed mechanism (blocking
  external HTTPS call at startup) and note the remote spec.kubeConfig
  exception that does not apply to cozystack in-cluster guest apiservers.
- Correct the test fixture comment: the proxy env is operator/patch-level, not
  installer-injected (the installer injects only KUBERNETES_SERVICE_*).

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit affdd70)
…sanitisation list

The BuildShardDeployment doc comment and the operator README are the only
record of how far the cloned shard container has diverged from flux-aio.
Add the two steps this fix introduced (dropping the inherited corporate-proxy
env and deriving a startupProbe) so the list stays the authoritative account
and does not invite re-adding what was deliberately removed.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 036a31b)
…ct alias regression

Model the source liveness probe as flux-aio actually ships it (a bare httpGet
with no timing fields) and tighten the startupProbe assertions:

- assert the complete normalised budget (delay/period/success/failure) instead
  of only FailureThreshold, so a later PeriodSeconds regression cannot silently
  restore a short crashloop window;
- assert TimeoutSeconds is inherited from the source probe rather than a
  hardcoded value, so forcing it stricter is caught;
- assert the startupProbe is a DeepCopy of liveness, not an alias, which would
  otherwise stamp the 30-failure startup budget onto liveness and produce a
  never-failing liveness probe.

The alias assertion is non-vacuous: it fails when the DeepCopy is regressed to
a plain alias.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit a10fcb3)
Name all six proxy env spellings in the sanitisation list, and replace
the unestablished 'external HTTPS call' / 'flux-aio does not restart'
claims with the grounded in-cluster mechanism: after the
KUBERNETES_SERVICE_HOST drop above, the shard talks to the management
apiserver over the kubelet-injected ClusterIP, which NO_PROXY=.svc does
not match, so that startup call is what stalls behind an unreachable
proxy. Comment-only; no behaviour change.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 66df3b2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants