fix(flux-shard-operator): repair sharded helm-controller crashloop behind an HTTP proxy - #3546

Approving — the design is right and it lands in the right place, as one more case in the existing sanitisation switch rather than a second mechanism alongside it.

I checked the load-bearing assumption that a standalone shard needs no external egress, and it holds for how this platform wires Flux. Chart delivery is chartRef: {kind: ExternalArtifact} throughout, and the artifact URLs advertise as flux.$(RUNTIME_NAMESPACE).svc (--storage-adv-addr); guest-cluster targets are in-cluster apiservers reached through the super-admin.svc kubeconfig key. Both were already NO_PROXY-bypassed, so dropping the proxy removes nothing that was working. Source-controller keeps its proxy env and keeps doing the fetching. The probe derivation is a genuine DeepCopy, so the 300s startup budget gates liveness without weakening it — a wedged running pod is still killed on the inherited 30s window.

One property worth stating in the PR description, because it decides whether anyone has to do anything by hand: already-crashlooping shards repair themselves on upgrade. containers[].env is listType=map keyed by name, the operator is the sole field manager applying with force ownership, and it previously owned the proxy keys — so dropping them from the apply set deletes them from the live object, and Recreate with one replica rolls the pod. No manual deletion needed.

Follow-ups, none of them blocking:

• The sanitisation list in the BuildShardDeployment doc comment enumerates every step (hostNetwork, single container, --events-addr, selector, --concurrent, SOURCE_*_LOCALHOST, KUBERNETES_SERVICE_*, tolerations, podAntiAffinity) and the README carries a shorter version of the same claim. Both should gain the two new entries — that list is the only record anywhere of how far the clone has diverged from flux-aio, so a stale one invites the next contributor to re-add what was deliberately removed.
• The root-cause wording is worth confirming before it settles as the reference explanation. Everything the shard dials advertises .svc; the one endpoint NO_PROXY=.svc never matched is the management apiserver, because this same function drops KUBERNETES_SERVICE_HOST and the shard falls back to the kubelet-injected ClusterIP, while flux-aio's hostNetwork KubePrism value is loopback and Go bypasses loopback regardless of NO_PROXY. If that is the mechanism, the fix here is unchanged, but "flux-aio is a latent risk if it ever restarts" would not be, and the residual exposure would sit in NO_PROXY at install time instead. An A/B with NO_PROXY set to the literal KUBERNETES_SERVICE_HOST value the pod actually has would settle it; the reported run with cluster CIDRs points the other way, which is exactly why it is worth pinning down.
• Two cheap test gaps: the assertion that the startup probe inherits TimeoutSeconds: 5 pins the fixture's value, but the real manifest probe is a bare httpGet with no timing fields at all, so a hardcoded sp.TimeoutSeconds = 5 would still pass — asserting against the source probe's own value would hold. And nothing currently fails if the DeepCopy regresses to an alias, which would put failureThreshold: 30 on liveness and give exactly the never-failing probe this change exists to avoid.
• Separate issue, and platform-level rather than this operator's: the freeze was invisible because a frozen HelmRelease is not an unready one. With nothing reconciling it, nothing writes status, so Ready keeps its last value indefinitely and HelmReleaseNotReady cannot see it. The exported metric set has no observedGeneration, no lastHandledReconcileAt, no timestamp of any kind, so nothing in the stack distinguishes a HelmRelease reconciled thirty seconds ago from one reconciled eight days ago — even though every one of them declares its own interval. That gap will hide the next reconciler stall just as completely, whatever causes it.

Broader note, explicitly not for this PR. Shards are currently derived by deep-copying the live flux-aio Deployment and subtracting what is flux-aio-specific, which buys free version-sync — and that matters more than it first appears, since flux-aio's own helm-controller is a peer bucket (--watch-label-selector=!sharding.fluxcd.io/key) and skew between it and the shards would mean two helm-controller versions reconciling one CRD version in one cluster. The cost is that the shard spec is defined by negation against a hostNetwork pod, so any field whose correctness depended on host networking is inherited until someone finds it in production: SOURCE_*_LOCALHOST, then KUBERNETES_SERVICE_*, now the proxy env. I would like the sharding controller to be able to stand on its own rather than being inseparable from the flux-aio topology. The version-sync requirement is real and should be kept, but it could be met by rendering the unsharded controller and the shards from one definition, instead of one being a mutated copy of the other's runtime object. Happy to open that as its own issue if there is appetite for it.

Two wording suggestions on the comments, nothing on the code — the implementation and the new tests both look right to me.

I checked the three new guards by introducing the regressions they describe: aliasing the liveness probe instead of deep-copying it, hardcoding TimeoutSeconds, and eroding PeriodSeconds to 1. All three fail the package, with the intended message each time. Rebuilding the fixture as a bare httpGet so it matches what flux-aio actually ships is a better fix than asserting around the old one.

Both suggestions below are about explanatory comments hardening around a mechanism that has not been confirmed yet. Neither affects behaviour, and neither is a reason to hold the change.