test(e2e): DO NOT MERGE — CI flake-hub parallel clone b - #3048

backupstrategy-controller creates the cozy-backups Bucket into the
tenant-root namespace, which cozystack-basics provisions. On a fresh
install the Bucket patch can race namespace creation and fail with
`namespaces "tenant-root" not found`, which then races the install
wait deadline and fails the run — on changes that have nothing to do
with backup ordering. Add the cozystack-basics dependsOn edge so the
target namespace exists first, pinned with a helm-unittest regression
guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

The reconciled, hasListenIPsForAllPods, fullReplication and healthy gates
polled status fields with a 60s ceiling, while the strictly-earlier
health.available gate had 300s. The operator stamps generations.reconciled
only after every process holds its role for minimumUptimeSecondsForBounce
(60s), making reconciled and the health fields that depend on it the last
status fields to converge. The short ceiling closed the gate seconds before
the field landed on a healthy, fully-reconciled cluster, causing intermittent
jsonpath poll timeouts.

Raise those four gates to 300s to match the health.available budget. The waits
stay event-driven; only the ceilings widen.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

Every *-application PackageSource that ships a system/*-rd component
renders an ApplicationDefinition (cozystack.io/v1alpha1). That CRD is
installed by the application-definition-crd component of
cozystack-engine. Without an explicit dependsOn edge, a fresh install
where Flux reconciles a *-application source before cozystack-engine
fails to render with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, which cascades into a broad install
timeout unrelated to whatever change triggered the reconcile.

Add `- cozystack.cozystack-engine` to the dependsOn of every source that
ships a system/*-rd component; gateway-application already carried the
edge for the TenantGateway CRD. Add a generic invariant test that fails
if any such source is missing the edge, so future sources cannot
reintroduce the race.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

The operator Deployment ships a single replica but always passes
--leader-elect to the manager. On one replica leader election adds no HA
value: a transient apiserver/etcd lease-renewal blip makes
controller-runtime self-terminate (os.Exit on "leader election lost"),
which crashloops the manager and surfaces at install time as a
Deployment stuck in Failed.

Gate --leader-elect on the replica count via a manager.replicaCount
value (default 1). The single-replica install now rides out
control-plane latency instead of crashlooping, and scaling to more than
one replica re-enables leader election automatically for genuine HA.

Also set the Deployment strategy to Recreate on a single replica: with
leader election off, a default RollingUpdate would briefly run two
managers (maxSurge) during an operator upgrade, both reconciling without
lease coordination. Recreate closes that window at zero availability
cost, while installs scaled past one replica keep RollingUpdate.

The change to the vendored opensearch-operator 2.8.0 chart is carried in
patches/leaderElection.diff and re-applied by 'make update' so it
survives a chart refresh.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

Under node CPU contention the linstor-controller JVM cannot answer its GET /health liveness probe within the piraeus-operator defaults (timeoutSeconds 1, failureThreshold 3). The kubelet then liveness-kills the container; the resulting crash-loop leaves the linstor-controller Service with no ready endpoints, linstor-csi gets EPERM, and RWX-Filesystem DataVolumes never bind, breaking tenant provisioning.

The operator-generated controller pod sets no resources (no CPU request, hence no scheduling/CPU guarantee) and ships the tight default probe. Add CPU/memory requests and relax the liveness timing via the controller podTemplate, which piraeus-operator merges into the Deployment as a strategic merge patch. The probe override carries only the timing fields; the httpGet target (GET /health on the api port) is left to the operator to avoid pinning a wrong path or scheme. Requests-only (no limits) keeps the JVM from being throttled or OOM-killed. The knobs are exposed under controller.* for tuning to observed RSS.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

The Pull Request Build job authenticates only to the push registry (OCIR),
so every base image (golang, ubuntu, alpine, nginx, ...) is pulled from
Docker Hub anonymously. Those pulls intermittently fail at
auth.docker.io/token under Docker Hub's anonymous-pull rate limit, leaving
the base image unresolved and failing make build.

Build on a docker-container buildx builder whose buildkitd routes docker.io
through mirror.gcr.io, Google's public pull-through cache for Docker Hub: no
auth, far higher limits, transparent fall-through to Docker Hub on a miss.
This reuses the builder pattern already proven in build-main.yaml (same
make build, same DOCKER_CONFIG auth forwarding); only the committed
hack/buildkitd.toml mirror config is new.

Fail-safe: if mirror.gcr.io is unreachable, BuildKit falls back to
registry-1.docker.io and the build still succeeds, so this strictly reduces
build failures without adding a new hard dependency.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…election

Every *-application source declares dependsOn cozystack.cozystack-engine so
its *-rd HelmRelease waits for the ApplicationDefinition CRD to register
before it reconciles. That is a universal install-ordering edge, not a
behavioral one.

The test-impact selector reused the same dependsOn graph to pick which
e2e app suites to run for a diff. Because the engine transitively depends
on postgres-operator, keycloak and cert-manager, a change to any of those
fanned out through the engine to every app, defeating test-impact analysis.

Exclude cozystack.cozystack-engine as a propagation hub in the reverse-dep
walk so the engine stays reachable but no longer pushes selection to its
app-dependents. A change to the engine itself still triggers the full suite
via the existing no-app-descendants safety net.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

The fluent-bit DaemonSet crash-loops on a loaded install, timing out its
HelmRelease and cascading into velero and backupstrategy-controller
DependencyNotReady failures. The subchart default liveness/readiness probes
(initialDelaySeconds 0, timeoutSeconds 1, failureThreshold 3) target the HTTP
server on :2020, but that server does not bind until AFTER the kubernetes
filter's 'testing connectivity with API server' step. Under a concurrent
install the kube-API can be unreachable for ~10s, delaying the :2020 bind,
while the single-threaded HTTP server can miss a 1s timeout when the engine is
busy. The probes then fire into a connection-refused port and liveness-kill
fluent-bit into a restart loop.

Give both probes load headroom (initialDelaySeconds 15, timeoutSeconds 5,
failureThreshold 6) so the API-gated late bind and busy-engine blips are
tolerated. The httpGet target is unchanged (path / on port http = :2020). Add
a helm-unittest pinning the relaxed timing and the preserved target.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…into ci/e2e-flake-hub

…ot-ordering' into ci/e2e-flake-hub

…ts' into ci/e2e-flake-hub

…on' into ci/e2e-flake-hub

…d-ordering' into ci/e2e-flake-hub

… ci/e2e-flake-hub

…ess' into ci/e2e-flake-hub

…' into ci/e2e-flake-hub

The E2E install intermittently fails on a CNI host->local-pod data-plane
transient: kubelet on a node reaches a local pod's readiness/startup probe
with "connection refused" for several minutes while overlay pod->pod
traffic works, then it self-heals. This is rooted in the cilium+kube-ovn
chaining config (forced enable-host-legacy-routing, CNI
InstallEndpointRoute:false), which delegates host->local-pod routing to
kube-ovn/ovn0. The crust-gather failure snapshot records Kubernetes object
state but not the node's L3 forwarding state, so the mechanism cannot be
root-caused after the fact.

On test failure, for each NotReady pod that is scheduled, has a podIP, and is
not terminal (Succeeded/Failed excluded), collect the host->pod data-plane
state from the pod's node into the crust-gather snapshot dir (so it ships in
the uploaded report):

- host netns via the kube-ovn cni-server (hostNetwork): ip route get <podIP>,
  ip neigh, ip rule, ip addr show ovn0, and the KERNEL conntrack entries for
  the podIP (conntrack -L, falling back to /proc/net/nf_conntrack). Under
  enable-host-legacy-routing the host->local-pod path traverses the kernel
  netfilter stack, not cilium BPF, so this is the authoritative conntrack
  table and route/rule state for the transient;
- cilium-agent on the node: endpoint list, bpf CT entries for the podIP, a
  bounded monitor --type drop, and hubble dropped verdicts when present
  (complementary: confirms the path is not handled in BPF);
- OVS/OVN on the node: ovs-ofctl dump-flows br-int, the OVN Port_Binding /
  Logical_Switch_Port for the pod, and a bounded ovn-sbctl lflow-list.

Diagnostic-only: it runs after the existing crust-gather collect and only on
an already-failed test, never mutates the cluster, and cannot change the
pass/fail outcome. Every live capture is time-boxed, every command is
|| true, the whole run has a wall-clock backstop, node-global captures are
deduped per node, and the pod count is capped. It no-ops cleanly when no
pods are affected or when a tool/pod is absent.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…ne' into hub-plus-diag

ovn-controller runs inside the ovs-ovn DaemonSet's openvswitch container,
sharing it with ovsdb-server and ovs-vswitchd. It is CPU-bound while
programming OpenFlow for newly created logical ports. Under a burst install
(many pods created at once) it competes for CPU in that single container and
CFS-throttles against the 2-core limit, lengthening flow-programming lag that
leaves freshly created pods unreachable from their node until the flows land.

Raise the ovs-ovn CPU limit from 2 to 4 cores to give ovn-controller headroom
to program flows promptly under burst, and lift the CPU request from 10m to
100m so it keeps a guaranteed share under node CPU contention. Memory is left
unchanged. Add a helm-unittest guard pinning the new CPU limit and request.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…ne' into hub-v2

…b-v2

…reuse

The cilium endpoint-leak healer's Tier-1 remedies (cilium-dbg endpoint
disconnect, or delete-reschedule the wedged pod) cannot clear a leak
that lives purely in the cilium-agent's in-memory state: when IPAM keeps
re-handing the SAME released IP, a delete-rescheduled pod re-wedges on
it, and on some variants the stale entry isn't even in the queryable
endpoint registry so disconnect no-ops. Both produce an inescapable
delete -> re-wedge / no-op loop that fails the install on
FailedCreatePodSandBox "IP ipv4:<X> is already in use" (cilium/cilium#38313).

Add a bounded Tier-2 escalation: track recurrence per (node,ip); once
the same IP survives ESCALATE_AFTER heal cycles of Tier-1, restart that
node's cilium-agent (delete its pod; the DaemonSet rebuilds it from
CRDs/IPAM without the stale entry), capped per node so it can never
loop-restart agents. A genuine duplicate-IP (live owner) is refused
before counting, so an agent restart never masks a real product bug.
Tier-1 stays the first remedy; escalation fires only after it proves
futile on the same IP.

RBAC needs no new verb: the existing cluster-wide pods get/list/delete
already covers deleting a cilium-agent pod in cozy-cilium.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…nto hub-v3

… hub-v4

The mariadb-operator hardcodes the mysqld-exporter liveness and readiness
probe as an httpGet on port 9104, which kubelet dials over the pod's IPv4
address. The exporter's default [::] (IPv6 wildcard) listener intermittently
refuses the IPv4 connect on single-stack-IPv4 data planes, so readiness fails
shortly after start, the pod flaps to NotReady, and the metrics Service keeps
the pod in notReadyAddresses — its endpoint never populates.

Pin the exporter to the IPv4 wildcard via --web.listen-address=0.0.0.0:9104 so
the kubelet probe connects deterministically. The operator appends user args to
its defaults and does not itself set --web.listen-address, so there is no
duplicate-flag conflict.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…sten' into hub-v4

…backoff

The master's /cluster/status:9333 handler calls topo.Leader(), which spins
an exponential backoff with MaxElapsedTime=20s before returning 200 whenever
the node has no leader in its view — e.g. a master that was briefly the initial
raft leader during bootstrap and then was demoted. The chart's master liveness
and readiness probes both httpGet /cluster/status with timeoutSeconds: 10, which
is shorter than that worst-case backoff. Such a master is then liveness-killed
roughly every 137s into a permanent CrashLoop, the master StatefulSet never
reaches full readiness, and the helm --wait install times out.

Raise both master probe timeouts to 30s so each probe outlasts the 20s backoff
(the volume component already uses 30s). The override sets only timeoutSeconds
and relies on Helm's deep-merge to keep the rest of the subchart probe defaults.
Both probes must be raised: leaving readiness at 10s would keep the master from
becoming Ready and still stall the StatefulSet.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

…imeout' into hub-v4