feat(e2e): consolidated E2E optimization changes (cozyreport + helmrelease-interval) - #2500

A 50MB tarball with 200 directories is opaque on first contact.
summary.txt at the root surfaces 'what is broken right now' — non-Ready
HRs, ImagePullBackOff pods, OOMKilled events, cert-manager state,
Flux Source health, storage binding, node pressure — so triage starts
with one file instead of a directory tree dive.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…0s in E2E

Adds --helmrelease-interval (default 5m, matching today's behaviour) to the
operator and a corresponding cozystackOperator.helmReleaseInterval value
on the cozy-installer chart (default empty -> no flag rendered ->
operator default 5m applies).

E2E install sets it to 30s. Rationale: with the 5m default, dependency-
blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only
every 5 min, producing an 8-10 min dead zone in the E2E install where
the fast pack of HRs is Ready but the slow tail hasn't been retried yet.
30s collapses the gap.

Production defaults are unchanged. The override is opt-in per install.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…tall

Previously these three configuration blocks ran as separate @test cases
after the 15m platform HR wait, adding ~1.5 min sequentially. None of
them gates platform HR reconcile, so they can run as a background prep
that completes during the main wait. The helper script polls for its
own prerequisites and is awaited at the end of the Install Cozystack
test so failures still surface deterministically.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…yment

The previous 5-min timeout on `until kubectl get deploy/linstor-controller`
fired during cold-install runs where the LINSTOR HR's dependency chain
(cert-manager → piraeus-operator-crds → piraeus-operator → linstor)
takes longer than 5 min to resolve, killing the whole post-install-prep
script via `set -eu`. With CI's 3x retry the failure was hidden; with the
retry removed it would surface every cold install.

Wait on the LINSTOR HR being Ready first (the actual prerequisite),
then keep the existence-poll as a near-instant backstop, then wait
for Available. Same final semantics, no fragile fixed-window timeout.

Surfaced by PR #2500 cozyreport diagnostics (post-install-prep.log).

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Across 5 sampled failure runs (30 PR runs total), 25/25 retry attempts
failed. The retry loop never recovered a flake — it only stretched
deterministic failures (e.g. kubernetes-previous: 11:37 + 7:49 + 10:16
= 29:42 wasted on one broken test). Replace with a single attempt plus
inline diagnostics (HR list + recent events) so the actual failure
surfaces immediately. Re-runs remain available via the standard
'gh run rerun' / empty-commit retry path.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

A failed cluster bootstrap or platform install is almost always a real
bug (Talos boot, chart, operator) — not a flake. The 3x retry on these
steps was the direct cause of the 250-min outlier run (24931612182):
each retry compounds 25-30 min of work, and the failure mode persists
across attempts. Single attempt + the existing 'collect-report' step
on always() gives faster signal and a debug archive.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Prepare environment is pure infrastructure (Talos image download, sandbox
VMs boot, network setup). Failures here are mostly noisy-runner /
transient infra hiccups (image-download stalls, NIC negotiation, etc.) —
not cozystack code under test. Retry-on-failure is the right policy for
infra setup.

Install Cozystack and Run E2E tests remain single-attempt: they exercise
cozystack code, where retries hide real bugs.

Refines the previous two retry-removal commits.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

The app bats files all share the pattern:

    kubectl apply -f - <<EOF ... EOF
    sleep 5
    kubectl wait hr <name> --for=condition=ready

The fixed sleep guesses how long the cozystack-operator needs to translate
the app CR into a HelmRelease. On a noisy CI runner or after a cold
install that wait is sometimes too short, so the immediately-following
`kubectl wait hr ...` fails because the HR object does not exist yet
(`kubectl wait` errors out instantly on a missing resource).

Replace each `sleep 5` (and the one outlier `sleep 15` in foundationdb
that was paying for the same risk) with an event-driven backstop that
polls for the HR's existence and exits the moment it appears, capped at
60s. The downstream `kubectl wait --for=condition=ready` is unchanged,
preserving the original total budget.

Same shape as the LINSTOR fix in commit eb87413 on feat/e2e-optimization:
wait for the actual prerequisite (the HelmRelease the operator creates),
not on a downstream artefact whose existence depends on it.

Files: postgres, mariadb, kafka, mongodb, clickhouse, redis, qdrant,
harbor, openbao, external-dns (both tests), vminstance (both tests, with
the existence backstop hoisted before the downstream vmi-ip poll), and
foundationdb.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…tl proxy

`kubectl proxy --port=21234 & sleep 0.5` guesses how long the proxy
needs to start listening. On a slow runner the curl that follows can
race the proxy and fail with "connection refused".

Replace the fixed `sleep 0.5` with `nc -z localhost 21234` polled until
the listener accepts a connection (10s cap). Same idiom already used in
hack/e2e-apps/bucket.bats for the seaweedfs-s3 port-forward.

Also save the proxy PID to a named variable so the trap kills the right
process even if a later background command lands first in $!.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Three poll/sleep sites in the install bats and the kubernetes runner are
NOT replaceable by `kubectl wait --for=condition=...` because the signal
they observe is not a Kubernetes API condition:

- e2e-install-cozystack.bats:55-56 - 5s pad lets late-arriving HRs join
  the awk-snapshot the parallel `kubectl wait` runs against. There is no
  k8s condition for "all expected platform HRs have been emitted" short
  of hard-coding the list.
- e2e-install-cozystack.bats:75 - LINSTOR node membership is reported by
  the linstor binary running inside the controller pod (kubectl exec),
  not a CRD status, so kubectl wait cannot subscribe to it.
- e2e-apps/run-kubernetes.sh:231-238 - validates the external HTTP path
  through MetalLB -> tenant ingress -> backend pod end-to-end. Not a
  single API condition.

Annotate each with TODO(e2e-replace-fixed-timeouts) and the rationale so
future readers do not "fix" them with a kubectl wait that does not work.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

The cozy-installer chart declares Namespace cozy-system itself, which
makes helm cold-install paradoxical:
  - WITH --create-namespace: helm pre-creates the ns without helm
    annotations; the chart's own Namespace apply then fails with
    'already exists'.
  - WITHOUT --create-namespace: helm fails before any apply because it
    can't write its release-secret to a non-existent ns.

The 3x retry was masking both directions — second attempt saw a partial
release and took the upgrade path, which patch-merges instead of
strict-create. Reproducible on every cold install across PRs.

Pre-create cozy-system with meta.helm.sh/release-name and
app.kubernetes.io/managed-by=Helm so helm adopts it cleanly during
install. Labels mirror the chart's template (PSA enforce=privileged etc.)
so behaviour matches what the chart would have done.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…hook

Removes the chart's `Namespace cozy-system` resource and replaces it with
a pre-install/pre-upgrade Job hook (cozy-system-labeler) that patches the
required labels onto the namespace after `--create-namespace` creates it.

Why: helm v3 has a known chicken-and-egg with charts that ship their own
Namespace:
  - WITH `--create-namespace` on the install command, helm pre-creates the
    namespace via plain kubectl-create (no helm meta annotations); the
    chart's own Namespace apply then fails with `already exists`.
  - WITHOUT `--create-namespace`, helm fails immediately because it cannot
    write its release-secret to a non-existent namespace.

Until now this was hidden by the 3x retry on `Install Cozystack` in
`.github/workflows/pull-requests.yaml`: first attempt always fails with
the conflict, second attempt sees the existing failed release and takes
the upgrade code path which patch-merges instead of strict-create.
Reproducible on every cold install. Surfaced cleanly when retries on the
install step were dropped.

After this change:
  - install commands use `helm upgrade --install --namespace cozy-system
    --create-namespace`. Standard pattern, matches kube-prometheus-stack /
    argo-cd / cert-manager / others.
  - the pre-install hook (SA + ClusterRole + ClusterRoleBinding + Job)
    patches `cozystack.io/system=true` and
    `pod-security.kubernetes.io/enforce=privileged` onto the namespace
    before main resources apply.
  - hook-delete-policy=before-hook-creation,hook-succeeded so the RBAC
    surface only exists during install/upgrade.

Verified end-to-end on a kind cluster: cold install in 3.3s, upgrade
idempotent, cleanup clean. Image pinned to `alpine/k8s:1.32.0` for the
hook (small, public, includes kubectl).

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Now that the chart fix (this same branch) bootstraps cozy-system via
--create-namespace + a pre-install label hook, the bats no longer needs
to pre-apply a hand-crafted helm-adoptable namespace. Restore plain
--create-namespace and drop the kubectl apply block.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…y timeouts

Two reliability fixes in `hack/e2e-apps/vminstance.bats`:

1. Delete-recreate race. Both @test blocks deleted the prior VMInstance/VMDisk
   with `kubectl ... --timeout=2m || true`. The `|| true` silently swallowed
   timeout errors, letting the test continue with the resource still in
   finalizer-drain. The next `kubectl apply` then no-ops with
   "Detected changes to resource ... which is currently being deleted",
   producing a NotFound on the downstream HR wait. Bumped the delete
   timeout to 3m and removed `|| true` so true delete failures surface
   loudly. End-of-test cleanup deletes also got the explicit timeout.

2. VM IP and VM ready timeouts. The 20s timeout for the VMI to acquire an
   IP was unrealistic for nested KubeVirt under runner load — virt-launcher
   + libvirt + cloud-init DHCP routinely takes 30-60s. Bumped to 120s with
   a 2s poll interval (60 polls instead of 4). VM ready timeout bumped from
   20s to 60s for the same reason.

Both surfaced as recurring first-attempt failures in the past 30 successful
PR runs (7/17 analysable runs in each case) — masked by the 3x retry on
`Run E2E tests`.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

… with configurable RetryInterval

The operator generated HelmReleases with `Install.Remediation{Retries: -1}`
and `Upgrade.Remediation{Retries: -1}`, which is the v2 way of saying
"retry forever, never remediate". Combined with the spec-level
`Interval: 5m`, this meant a failed install/upgrade waited a full 5
minutes before the next attempt — even when the underlying race that
caused the failure (e.g. a chart artifact not quite Ready, a HR
dependency reconciling out of order) cleared in seconds.

Audit of the last 30 successful PR runs found that every single run had
its `Install Cozystack` step's `kubectl wait hr/seaweedfs-system` time
out at the 2-minute mark and recover only after an inline
`flux reconcile --force` workaround. Same root cause: 5-minute retry
interval was longer than the test's wait window. That workaround can be
removed once this change lands.

Switch to `Strategy.Name=RetryOnFailure` with `RetryInterval` exposed via
a new `--helmrelease-retry-interval` operator flag (default 30s):

- Functionally equivalent to the previous configuration ("retry forever
  on failure") since `Retries: -1` meant remediation never fired anyway.
- Decouples retry-on-failure timing from `spec.Interval` so failed
  releases recover at 30s without polling healthy releases at the same
  cadence (which would multiply controller load by ~10x for no benefit).
- Helm chart exposes `cozystackOperator.helmReleaseRetryInterval`,
  defaulting to empty (operator uses its own 30s default).
- Same flag pattern as the existing `--helmrelease-interval` (5m default),
  also added in this PR via cherry-pick.

Verified: go build green, go test ./internal/operator/... pass, helm
template renders both flags conditionally on values.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

The HelmRelease generation in package_reconciler hardcoded three values
that have caused recurring pain:

- Install.Timeout = 10m (line 214)
- Upgrade.Timeout = 10m (line 220)
- MaxHistory = unset (Helm default 5)

The 10m timeout in particular contradicts the per-Application timeout
work done in `pkg/config/config.go` and
`pkg/registry/apps/application/rest.go` (commit 7b146cb), which lets
individual Applications override the install/upgrade timeout via
annotation. The PackageSource side had no equivalent — etcd works around
it by hand-rolling its HR YAML in `packages/apps/tenant/templates/etcd.yaml`
to set a 30m timeout, harbor has had multiple commits chasing the same
problem from the other side. This closes that gap.

New operator flags (each maps to a chart value with empty default;
operator uses its own default when value is empty):

- `--helmrelease-install-timeout` (default 10m) → Spec.Install.Timeout
- `--helmrelease-upgrade-timeout` (default 10m) → Spec.Upgrade.Timeout
- `--helmrelease-max-history` (default 5)        → Spec.MaxHistory

Production behaviour unchanged. E2E and edge cases (cert rotation,
slow-installing charts) can override per-cluster without modifying
chart manifests.

Per-component overrides on `ComponentInstall` (mirroring the existing
`UpgradeCRDs` pattern) would be a strictly better fix for charts like
etcd that need an outlier timeout — left as a deliberate followup since
it touches the Package CR API.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…ule nodes

The pre-install Job runs before any CNI is installed, so all nodes are
tainted NotReady-NoSchedule and the pod has no pod network. Without
adjustment the Job's pod sits Pending until helm's pre-install timeout
fires, blocking the entire install.

Surfaced in PR #2500 CI run 25040584552 attempt 2:
  FailedScheduling  0/3 nodes are available: 3 node(s) had untolerated
  taint {node.kubernetes.io/not-ready: }.

Mirror the cozystack-operator deployment's scheduling pattern:
- hostNetwork=true so the pod doesn't depend on CNI
- Tolerations matching the operator
  (not-ready / unreachable / cilium.agent-not-ready / cloudprovider.uninitialized)
- Variant-aware KUBERNETES_SERVICE_HOST/PORT env so kubectl reaches the
  apiserver before kube-proxy + CNI are up:
    talos:   localhost:7445   (KubePrism)
    generic: cozystack.apiServerHost / Port
    hosted:  default in-cluster

Verified on the sandbox kind cluster that the rendered Job manifest
schedules and the kubectl container starts (the kind cluster itself
doesn't run KubePrism so the env-var path can't be end-to-end-tested
there; the Talos E2E will exercise it).

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

The chart fix (3a87485, be0a668) removes the chart-defined Namespace
without replacing it. On the Talos E2E cluster the apiserver enforces
PodSecurity baseline-by-default on bare namespaces, so the cozystack-
operator pod (hostNetwork=true) is rejected if cozy-system is created
without enforce=privileged.

Apply the namespace from the bats with the labels the operator depends
on, then run helm install without --create-namespace. Mirrors the
documented production install procedure for the new chart shape.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

`kubectl wait <kind> <name>` errors immediately with NotFound if the
resource doesn't exist yet — even with --for=condition or --for=jsonpath.
The redis test in CI run 25043350705 failed in 1 second for exactly this
reason: the redis-failover operator hadn't created the PVC by the time
the test waited for it. Previously the 3x retry on `Run E2E tests`
masked this race; with retry dropped, every such call is a flake risk.

Add a small `until kubectl get` existence backstop before each
kubectl wait, matching the pattern already established for HRs in
commit 66888c9.

33 backstops across 12 files:
  bucket.bats         — bucketclaims, bucketaccesses x2
  clickhouse.bats     — statefulset 0-0 (0-1 already covered)
  harbor.bats         — deploy x3, bucketclaims, bucketaccesses
  kafka.bats          — kafkas
  mariadb.bats        — statefulset, deployment
  mongodb.bats        — statefulset
  openbao.bats        — sts, pvc
  postgres.bats       — job.batch
  qdrant.bats         — sts, pvc
  redis.bats          — pvc, deploy, sts (the trigger)
  vminstance.bats     — dv, pvc, vm (with 120s for KubeVirt latency)
  e2e-install-cozystack.bats — apiservices, sts/etcd, vmalert,
                                vmalertmanager, vlclusters, vmcluster,
                                clusters.postgresql.cnpg.io,
                                deploy/grafana-deployment, namespace

Same pattern, same 60s default timeout for the existence wait (120s for
nested-virt resources). Once the resource exists, the original wait
timeout takes over.

Run-kubernetes.sh has the same race shape on several waits (nfs, kamaji,
machinedeployment, etc.) — out of scope here; flagged for follow-up.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Two bugs caused the harbor E2E to fail (CI #25049445125):

1. cozy-harbor's bucket-secret.yaml called `index $existingSecret.data
   "BucketInfo"` unconditionally. During first-time install the COSI
   BucketAccess controller may create the credentials Secret as a
   placeholder before populating it, so `.data` is nil and the chart
   render crashes with `index of untyped nil`.

2. The downstream `<release>-system` HelmRelease starts reconciling
   immediately, in parallel with bucket provisioning, hitting bug #1
   on every retry within the test's 5-minute window.

The previous shape relied on `lookup` to read the Secret at render
time. helm-controller's upgrade trigger is digest-based over composed
values + chart artifact, so a `lookup` returning new data on a later
reconcile is not enough on its own to force an upgrade — the rendered
output may diverge but the digest does not.

Switch to a values-driven shape:

- `bucket-secret.yaml` now reads `.Values.bucket.bucketInfo` (a JSON
  string) and uses `dig` for safe access; if the value is empty or the
  expected nested fields are missing, no `*-registry-s3` Secret is
  rendered.
- The `<release>-system` HR sources `BucketInfo` via `valuesFrom` with
  `targetPath: bucket.bucketInfo`. With the default `optional: false`,
  helm-controller will refuse to compose values until the key exists,
  which both gates initial reconciliation on the BucketAccess Secret
  being populated and forces a config-digest change (and thus a helm
  upgrade) when its contents change.
- `bucket.secretName` is removed from the system chart's values and
  from the apps chart's `values:` block; the only consumer (the
  `lookup` call) is gone.

Verified four scenarios via `helm template` against the system chart:
unset bucketInfo, empty string, empty JSON object `{}`, and a
fully-populated BucketInfo — only the last renders the registry-s3
Secret; the others render nothing without erroring.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

… BucketAccess reconcile

Upstream COSI v0.2.2's BucketAccess reconciler does a Get->mutate->Update
on the parent Bucket and surfaces "Operation cannot be fulfilled ... the
object has been modified" as a FailedGrantAccess event when it races
against the Bucket reconciler in the same controller process. Wrap the
mutation in retry.RetryOnConflict so the reconcile loop refreshes and
retries instead of leaking the conflict to users.

Carried as 91-bucketaccess-conflict-retry.diff until upstreamed
(cf. 89-reconciliation.diff and 90-bucket-name.diff, both dropped
in c29d501 once merged upstream).

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Some workloads (OVN raft, LINSTOR controller) fail when replicas start
at different times due to image-pull stagger across nodes. Add a
DaemonSet-based pre-pull step that runs before helm install, ensuring
all nodes have the images cached so every replica starts within
milliseconds of each other.

Covers kube-ovn (confirmed OVN raft race), piraeus-server, and
linstor-csi. Comments in hack/e2e-prepull-images.sh point to the
source chart values so versions stay in sync.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>