ci: consolidate CI fixes (#2602 flux 2.8 + #2601 seaweedfs-db + #2612 kubevirt TPM + #2615 e2e trace) - #2619

…or-vpa HR

The packages/system/vertical-pod-autoscaler chart's vpa-for-vpa.yaml template
emits an HR in cozy-vpa-for-vpa that the parent helm install applies and waits
on. That nested HR had `dependsOn: monitoring-agents`, which never resolves:

  parent VPA helm install
    waits for nested vpa-for-vpa HR Ready
      waits for monitoring-agents HR Ready
        waits for cozystack.vertical-pod-autoscaler PackageSource Ready
          waits for parent VPA HR Ready  <-- back to start

Symptom observed on PR #2619 e2e (cozyreport):

  Helm upgrade failed for release cozy-vertical-pod-autoscaler/
  vertical-pod-autoscaler with chart cozy-vertical-pod-autoscaler@...:
  timeout waiting for: [HelmRelease/cozy-vpa-for-vpa/vpa-for-vpa
  status: 'InProgress']

The recommender's /health-check readiness probe is served by the binary
itself and doesn't require Prometheus. Removing the dependsOn lets the
nested HR become Ready quickly; the recommender will log connection
errors against vmselect until monitoring-agents installs later in the
boot sequence, then start producing recommendations normally.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…or-vpa HR

The packages/system/vertical-pod-autoscaler chart's vpa-for-vpa.yaml template
emits an HR in cozy-vpa-for-vpa that the parent helm install applies and waits
on. That nested HR had `dependsOn: monitoring-agents`, which never resolves:

  parent VPA helm install
    waits for nested vpa-for-vpa HR Ready
      waits for monitoring-agents HR Ready
        waits for cozystack.vertical-pod-autoscaler PackageSource Ready
          waits for parent VPA HR Ready  <-- back to start

Symptom observed on PR #2619 e2e (cozyreport):

  Helm upgrade failed for release cozy-vertical-pod-autoscaler/
  vertical-pod-autoscaler with chart cozy-vertical-pod-autoscaler@...:
  timeout waiting for: [HelmRelease/cozy-vpa-for-vpa/vpa-for-vpa
  status: 'InProgress']

The recommender's /health-check readiness probe is served by the binary
itself and doesn't require Prometheus. Removing the dependsOn lets the
nested HR become Ready quickly; the recommender will log connection
errors against vmselect until monitoring-agents installs later in the
boot sequence, then start producing recommendations normally.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

PR #2619 split seaweedfs into a serialized chain
(seaweedfs-db -> seaweedfs-system -> seaweedfs) that needs ~5 min of
wall-clock to install: CNPG bootstrap, master quorum, then the wrapper.
With the prior 5m wait budget on the parent tenant HR, helm-controller
would time out before the children reach Ready, rollback would delete
the child HRs, and the upgrade would retry in a loop. The e2e bats step
that patches the Tenant CR and waits 60s for child HRs to exist landed
in the rollback gap and timed out.

Two complementary changes:

- tenant-root is created statically by cozystack-basics, so its
  HelmRelease spec.timeout is bumped 5m -> 15m directly.
- Sub-tenants (Tenant CRs) get their HR built by cozystack-api from
  the tenant ApplicationDefinition. Add the
  release.cozystack.io/helm-install-timeout=15m annotation that
  kubernetes-rd already uses to opt the kind into the longer budget.

Keeps both values in lockstep and documents the coupling inline.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

The bats step had two waits for tenant-root child HRs:

  kubectl wait hr/etcd hr/ingress hr/tenant-root hr/seaweedfs --timeout=4m
  kubectl wait hr/monitoring hr/seaweedfs-system --timeout=2m

The second line was the residue of a flux-reconcile-force workaround
removed in 7d292eb; the mechanical cleanup kept the same HRs on a
separate 2m wait without revisiting whether the line still made sense.
After PR #2509 fixed the underlying timing bug:

- monitoring is a top-level tenant-root child, identical shape to etcd
  and ingress; its Ready is already implied by hr/tenant-root Ready on
  the first line.
- seaweedfs-system is an internal HR of the seaweedfs umbrella chart;
  its Ready is already implied by hr/seaweedfs Ready on the first line.

So the second wait was always tautologically true by the time it ran,
and the naming inconsistency (top-level monitoring next to internal
seaweedfs-system) was an accident of the historical workaround. Promote
monitoring to the first line where it belongs and drop the redundant
second wait.

Also bump the wait from 4m to 10m. seaweedfs now installs as a serial
chain seaweedfs-db (CNPG bootstrap) -> seaweedfs-system (raft quorum)
-> seaweedfs wrapper, so the parent tenant-root HR's Ready flip lags
~5-6 min. The previous 4m budget was sized for the old single-HR
seaweedfs and is the immediate cause of the e2e failure on PR #2619.
The 10m budget stays inside tenant-root HR.spec.timeout (15m).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Flux helm-controller v1.5.0 (shipped in Flux v2.8) replaced the legacy
Helm readiness logic with kstatus-based health checking, which polls
Deployment/StatefulSet/Service status and waits for kstatus to assess
rollout-ready. This typically adds 10-30s of latency between resource
apply and HR condition Ready=True compared to v2.7.

Our per-app bats tests had `kubectl wait hr/... --for=condition=ready
--timeout=` budgets of 20s, 30s, 60s, 100s — sized for v2.7's faster
readiness flip and previously masked by the 3x retry on Run E2E tests
(dropped in #2558). Under v2.8 with no retry, every app that runs after
the first few hit its HR wait timeout exactly, surfacing as 13 of 16
test failures on PR #2619.

Standardise on 5m for HR Ready across all app tests. Tight enough to
catch genuine bring-up failures within the per-app job budget, loose
enough to absorb kstatus dispatch jitter on a management cluster that
is concurrently reconciling multiple tenant-test HRs.

Post-HR-Ready checks for app-specific resources (StatefulSet/Service/
PVC/operator CRs) remain — for CR-based apps kstatus only observes
that the chart applied, not that the operator finished. Audit of
which of those are actually redundant under v2.8 kstatus is queued
as a follow-up.

Also bumps the 10s "kamajicontrolplane appears" wait in
run-kubernetes.sh to 2m, and the kubernetes-addon HR Ready loop from
1m to 5m for the same reason.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

Five PackageSources whose charts apply cert-manager Certificate CRs and
mount the resulting Secret on a webhook Deployment lacked an explicit
dependsOn: cozystack.cert-manager. Under Flux v2.8 helm-controller v1.5.0
kstatus, the dependent Deployment trips progressDeadlineSeconds (10m
default) when cert-manager controller isn't running yet, and the HR is
failed early. Repeatedly observed on PR #2619 with capi-operator,
kamaji, rabbitmq-operator, cozystack-api (cozystack-engine), and
ingress-nginx all cascading to InstallFailed at ~12-15m into a fresh
install.

The mariadb-operator, kubeovn-webhook, snapshot-controller, etcd-operator,
postgres-operator, mongodb-operator, victoria-metrics-operator, linstor,
and linstor-scheduler PackageSources already had the dependency, which
is why they weren't in the failure set.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

…afka WorkloadMonitor replicas paths

Flux v2.8 / helm-controller v1.5.0 embeds Helm v4, which renamed --force
to --force-replace and rejects combining it with SSA's --force-conflicts.
Existing HelmReleases with spec.upgrade.force: true fail every upgrade with
"cannot use force conflicts and force replace together", looping
upgrade -> rollback -> upgrade and blocking dependants. Observed on PR #2619
CI for tenant-root/{ingress-nginx-system,seaweedfs-db}, tenant-test/info,
and others.

Per Flux docs, upgrade.force has been a no-op since v2.8 anyway, so removing
it is safe on green clusters and unblocks broken ones.

Also fix packages/apps/kafka/templates/workloadmonitor.yaml: it referenced
.Values.replicas (no such field; values.yaml has kafka.replicas and
zookeeper.replicas). Flux v2.7 silently dropped the null; v2.8 SSA forwards
it, and the WorkloadMonitor CRD rejects "spec.replicas: null" as not
type:integer, which blocks every kafka-test install.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>

)

## What this PR does

Adds `startupProbe` to two operator deployments whose readiness probes
fail repeatedly during a cold cluster install, generating noise and (for
CDI) blocking dependants on slow leader-election:

- **`kubevirt-cdi-operator`** — leader-election against a slow
`kube-apiserver` (Kamaji control plane bringup) consistently takes >30s
on a cold install. The default readinessProbe with 1s `periodSeconds` /
3 `failureThreshold` fires the first time election times out, the
operator pod gets marked NotReady, and dependants block on it.
- **`grafana-operator`** — bootstrap CRD discovery + Helm-managed
Dashboard reconcile loop spikes startup duration past readinessProbe's
default budget under load.

`startupProbe` gives each operator a longer one-time window to reach
steady state; the regular `readinessProbe` then takes over with its
tight per-second cadence to detect genuine ongoing failures.

## Origin

Both commits lifted unchanged from #2619.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Configuration Updates**
  * Added startup health checks to Grafana operator container
  * Added startup health checks to KubeVirt CDI operator container

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2725?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

## What this PR does

Pre-pulls every image referenced by `packages/system/kubeovn`,
`packages/system/linstor`, and `packages/system/cert-manager` onto every
node before those HelmReleases install. Cluster-member workloads (OVN
raft, LINSTOR, cert-manager webhook) fail when replicas start at
different times due to per-node image-pull stagger; pre-pulling means
all replicas start with images already cached.

## Mechanics

Renders each chart with `helm template`, then runs a `yq` filter that
walks every PodSpec-shaped object and emits the images of each container
— scoping the output to images the kubelet actually pulls (skips
configmap fields and CRD examples that happen to contain an `image:`
key). Each render is staged through a tmp file so a helm-template
failure trips `set -e` cleanly (cozytest.sh runs `@test` bodies under
`/bin/sh`, which is dash on Ubuntu CI — no `pipefail`).

The pre-pull DaemonSet itself is the hard part: distroless images
(cert-manager and friends) ship no shell and no `/bin/sleep`, so the
obvious `command: ["sleep", "infinity"]` fails with `exec: ENOENT`. The
script stages a statically-linked `busybox:musl` into a shared
`emptyDir` from an initContainer; every prepull container then execs
`/shared/sleep` regardless of what the image itself ships. The glibc
`busybox` tag is unusable — it needs `/lib64/ld-linux-x86-64.so.2` which
distroless lacks.

## Origin

Lifted from #2619 — squash of `test(e2e): pre-pull cert-manager images`
→ POSIX rewrite → cert-manager removal (failed on distroless) →
reintroduction with the busybox helper that solves the distroless
problem. Final state only — the four iterative commits aren't
review-friendly individually.

## Verification

- `sh -n hack/e2e-prepull-images.sh` clean
- All chart-referenced images held Ready under the DaemonSet on local
kind v1.33.1 single-node

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Improved end-to-end test reliability for platform image pre-pulling by
rendering charts separately and consolidating extracted images.
* Simplified and consolidated readiness waits during tenant
configuration for more robust application readiness checks.

* **Chores**
* Enhanced image pre-pulling to reliably handle distroless images using
a staged helper binary and shared volume approach.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2724?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Five PackageSources whose charts apply cert-manager Certificate CRs and
mount the resulting Secret on a webhook Deployment lacked an explicit
dependsOn: cozystack.cert-manager. Under Flux v2.8 helm-controller v1.5.0
kstatus, the dependent Deployment trips progressDeadlineSeconds (10m
default) when cert-manager controller isn't running yet, and the HR is
failed early. Repeatedly observed on PR #2619 with capi-operator,
kamaji, rabbitmq-operator, cozystack-api (cozystack-engine), and
ingress-nginx all cascading to InstallFailed at ~12-15m into a fresh
install.

The mariadb-operator, kubeovn-webhook, snapshot-controller, etcd-operator,
postgres-operator, mongodb-operator, victoria-metrics-operator, linstor,
and linstor-scheduler PackageSources already had the dependency, which
is why they weren't in the failure set.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 0fc0796)

The dashboard gatekeeper (oauth2-proxy) does OIDC discovery against the
external host at container startup. The e2e sandbox uses
host=example.org which does not resolve, so the auth-proxy crashloops:

  Get "https://keycloak.example.org/realms/cozy/.well-known/openid-configuration":
    dial tcp: lookup keycloak.example.org: no such host

Under Flux v2.8 kstatus the gatekeeper Deployment then flips to
'Failed' (rather than the pre-v2.8 'InProgress') and stalls the
cozy-dashboard/dashboard HelmRelease, which in turn blocks
cozy-fluxcd/flux-plunger via dependsOn. Previous runs masked this with
an unrelated ImagePullBackOff on cozy-dashboard-console; with the
console image fixed the OIDC discovery failure now surfaces.

packages/core/platform/values.yaml:194-198 already declares
authentication.oidc.keycloakInternalUrl. When set, the dashboard
chart (packages/system/dashboard/templates/gatekeeper.yaml:59-66)
renders --skip-oidc-discovery plus explicit redeem/jwks/validate/
backend-logout URLs pointing at the in-cluster keycloak Service.
Browser-facing redirect-url and login-url stay on the external host
(they are not exercised by the e2e flow).

The patch sets the value alongside oidc.enabled in the same
kubectl-patch so the configmap and gatekeeper rollout happen in one
reconcile pass.

Observed on CI run 26142986969 (PR #2619).

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5915a57)

…2726)

## What this PR does

Adds `cert-manager` as a `dependsOn` entry to the five PackageSources
whose charts apply `cert-manager.io/v1` `Certificate` CRs and need the
webhook ready before their first reconcile. Without this, the helm
install renders Certificate manifests, the cert-manager webhook isn't
admitting requests yet (cold cluster), the apply fails, and
helm-controller retries until the webhook eventually comes up — burning
~2-3 minutes per affected package on cold installs.

The five PackageSources: `capi-operator`, `cozystack-engine`,
`ingress-nginx`, `kamaji`, and `rabbitmq-operator` (the platform
`cozy-system` set with webhook-cert consumers; not the apps layer).

In `cozystack-engine`, the dependency is added to both the `default` and
`oidc` variants — the `oidc` variant installs the same `cozystack-api`
and `lineage-controller-webhook` components that render Certificate
resources, so it had the same race.

## Origin

First commit lifted unchanged from #2619. Second commit (`oidc` variant)
added in response to review feedback.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **Chores**
* Updated package source dependencies across multiple components to
ensure proper installation sequencing.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2726?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…rl (#2728)

## What this PR does

Bundles four independent e2e improvements lifted from #2619:

1. **`test(e2e): fold ouroboros assertions into kubernetes-latest`** —
moves the standalone `hack/e2e-apps/ouroboros.bats` (which provisioned
its own full Kamaji control plane, ~25m) into the existing
`kubernetes-latest` run as an optional addon. The assertions cover
ouroboros HR Ready, controller pod Running, `Ingress → coredns-custom`
rewrite-line injection, and end-to-end DNS resolution from inside the
tenant. Reuses the existing tenant cluster instead of building a second
one.

2. **`test(e2e): wait for tenant CSI HR Ready before NFS test`** —
explicit `kubectl wait hr/kubernetes-${test_name}-csi
--for=condition=ready` ahead of the NFS pod. Otherwise the NFS pod
schedules while the `kubevirt-csi-node` DaemonSet is still rolling out,
eats ~1m on `FailedAttachVolume` retries, and trips the pod-Succeeded
wait.

3. **`test(e2e): bump NFS pod wait to 10m to fit slow-path chain`** —
was 5m. With dependencies properly serialized (per #2), 5m was tight on
slow runners; 10m gives the chain genuine room without masking real
bugs.

4. **`test(e2e): set keycloakInternalUrl when enabling OIDC`** — the
dashboard gatekeeper (oauth2-proxy) does OIDC discovery against the
keycloak Service URL; without an explicit `keycloakInternalUrl`, it
falls back to the public ingress URL which isn't routable from inside
the cluster yet during install.

## Origin

All four commits lifted unchanged from #2619.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Integrated Ouroboros validations into the main Kubernetes end-to-end
flow (removed separate standalone Ouroboros test)
* Added hairpin-NAT and in-tenant DNS resolution checks for Ouroboros
scenarios
* Improved Keycloak OIDC test by configuring an in-cluster internal URL
* Increased various timeouts and readiness waits and added tenant CSI
readiness gating to improve test reliability

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2728?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…afka WorkloadMonitor replicas paths

Flux v2.8 / helm-controller v1.5.0 embeds Helm v4, which renamed --force
to --force-replace and rejects combining it with SSA's --force-conflicts.
Existing HelmReleases with spec.upgrade.force: true fail every upgrade with
"cannot use force conflicts and force replace together", looping
upgrade -> rollback -> upgrade and blocking dependants. Observed on PR #2619
CI for tenant-root/{ingress-nginx-system,seaweedfs-db}, tenant-test/info,
and others.

Per Flux docs, upgrade.force has been a no-op since v2.8 anyway, so removing
it is safe on green clusters and unblocks broken ones.

Also fix packages/apps/kafka/templates/workloadmonitor.yaml: it referenced
.Values.replicas (no such field; values.yaml has kafka.replicas and
zookeeper.replicas). Flux v2.7 silently dropped the null; v2.8 SSA forwards
it, and the WorkloadMonitor CRD rejects "spec.replicas: null" as not
type:integer, which blocks every kafka-test install.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 7e286cf)

…or-vpa HR

The packages/system/vertical-pod-autoscaler chart's vpa-for-vpa.yaml template
emits an HR in cozy-vpa-for-vpa that the parent helm install applies and waits
on. That nested HR had `dependsOn: monitoring-agents`, which never resolves:

  parent VPA helm install
    waits for nested vpa-for-vpa HR Ready
      waits for monitoring-agents HR Ready
        waits for cozystack.vertical-pod-autoscaler PackageSource Ready
          waits for parent VPA HR Ready  <-- back to start

Symptom observed on PR #2619 e2e (cozyreport):

  Helm upgrade failed for release cozy-vertical-pod-autoscaler/
  vertical-pod-autoscaler with chart cozy-vertical-pod-autoscaler@...:
  timeout waiting for: [HelmRelease/cozy-vpa-for-vpa/vpa-for-vpa
  status: 'InProgress']

The recommender's /health-check readiness probe is served by the binary
itself and doesn't require Prometheus. Removing the dependsOn lets the
nested HR become Ready quickly; the recommender will log connection
errors against vmselect until monitoring-agents installs later in the
boot sequence, then start producing recommendations normally.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 08433dc)

Flux helm-controller v1.5.0 (shipped in Flux v2.8) replaced the legacy
Helm readiness logic with kstatus-based health checking, which polls
Deployment/StatefulSet/Service status and waits for kstatus to assess
rollout-ready. This typically adds 10-30s of latency between resource
apply and HR condition Ready=True compared to v2.7.

Our per-app bats tests had `kubectl wait hr/... --for=condition=ready
--timeout=` budgets of 20s, 30s, 60s, 100s — sized for v2.7's faster
readiness flip and previously masked by the 3x retry on Run E2E tests
(dropped in #2558). Under v2.8 with no retry, every app that runs after
the first few hit its HR wait timeout exactly, surfacing as 13 of 16
test failures on PR #2619.

Standardise on 5m for HR Ready across all app tests. Tight enough to
catch genuine bring-up failures within the per-app job budget, loose
enough to absorb kstatus dispatch jitter on a management cluster that
is concurrently reconciling multiple tenant-test HRs.

Post-HR-Ready checks for app-specific resources (StatefulSet/Service/
PVC/operator CRs) remain — for CR-based apps kstatus only observes
that the chart applied, not that the operator finished. Audit of
which of those are actually redundant under v2.8 kstatus is queued
as a follow-up.

Also bumps the 10s "kamajicontrolplane appears" wait in
run-kubernetes.sh to 2m, and the kubernetes-addon HR Ready loop from
1m to 5m for the same reason.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5ef02ee)

PR #2619 split seaweedfs into a serialized chain
(seaweedfs-db -> seaweedfs-system -> seaweedfs) that needs ~5 min of
wall-clock to install: CNPG bootstrap, master quorum, then the wrapper.
With the prior 5m wait budget on the parent tenant HR, helm-controller
would time out before the children reach Ready, rollback would delete
the child HRs, and the upgrade would retry in a loop. The e2e bats step
that patches the Tenant CR and waits 60s for child HRs to exist landed
in the rollback gap and timed out.

Two complementary changes:

- tenant-root is created statically by cozystack-basics, so its
  HelmRelease spec.timeout is bumped 5m -> 15m directly.
- Sub-tenants (Tenant CRs) get their HR built by cozystack-api from
  the tenant ApplicationDefinition. Add the
  release.cozystack.io/helm-install-timeout=15m annotation that
  kubernetes-rd already uses to opt the kind into the longer budget.

Keeps both values in lockstep and documents the coupling inline.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit cc9da3a)

…edfs-system (#2601)

## What this PR does

Splits the `seaweedfs-system` HelmRelease into two: the CNPG
`Cluster/seaweedfs-db` moves to its own HR (`seaweedfs-db`), and the
application HR (`seaweedfs-system`) `dependsOn` it. The new HR uses Flux
v2 `HelmRelease.spec.healthCheckExprs` with a CEL expression on
`Cluster.status.conditions[type=Ready]` plus `waitStrategy.name:
poller`, so its `Ready=True` only flips after the postgres primary is
actually serving connections — not just after `helm install` applied the
`Cluster` CR.

This eliminates the `seaweedfs-filer` CrashLoopBackOff race on a fresh
tenant install. With Cilium `kubeProxyReplacement: true`, socket-LB
returns `EPERM` from `connect(2)` to ClusterIPs with no Ready endpoints.
Pre-split, the filer StatefulSet scheduled concurrently with the CNPG
bootstrap (~55–70s of unavailable postgres), each `connect()` failed
`EPERM`, kubelet exponential restart backoff pushed past the e2e bats
`kubectl wait hr/seaweedfs-system --timeout=2m` window, and the
"Configure Tenant and wait for applications" test failed.

## Stacked on #2602

This PR depends on **#2602** (Flux v2.8.0 upgrade) — `healthCheckExprs`
is a helm-controller v1.5.0 feature. GitHub base set to `flux-bump-v048`
so the diff cleanly shows only this PR's changes; will auto-rebase onto
`main` once #2602 merges.

## Commits

1. **`refactor(seaweedfs)`** — the split itself. Squash of the three
iterative commits from the consolidated branch (#2619): the initial
split, a `dependsOn` merge-collision fix (collapses unconditional
`seaweedfs-db` dependency with the namespace-guarded ingress dependency
into one list), and the `healthCheckExprs` three-predicate form (route
CNPG's `ClusterIsNotReady` bootstrap condition to `inProgress`, reserve
`failed` for other `Ready=False` reasons, lean on the HR `timeout: 10m`
as the real backstop).
2. **`fix(tenant)`** — bump parent HR `spec.timeout` from 5m → 15m so
the serialized `seaweedfs-db → seaweedfs-system → seaweedfs` chain has
enough wall-clock budget. Two complementary changes: the static
`tenant-root` HR (created by `cozystack-basics`) gets its `spec.timeout`
bumped directly; sub-tenants (Tenant CRs whose HR is built by
`cozystack-api` from the `tenant` ApplicationDefinition) get the
`release.cozystack.io/helm-install-timeout=15m` annotation that
`kubernetes-rd` already uses to opt into the longer budget.
3. **`test(e2e)`** — consolidate two redundant child-HR waits in
`hack/e2e-install-cozystack.bats` into one budgeted assert. The second
`kubectl wait` (`hr/monitoring hr/seaweedfs-system --timeout=2m`) was a
residue of a `flux-reconcile-force` workaround removed in #2509;
`monitoring`'s Ready is already implied by `hr/tenant-root` Ready and
`seaweedfs-system`'s by `hr/seaweedfs` Ready, so the line was
tautologically true by the time it ran.

## Migration

`packages/core/platform/images/migrations/migrations/42` adopts existing
`Cluster/seaweedfs-db` resources into the new release on upgrade —
rewrites `meta.helm.sh/release-name: seaweedfs-system → seaweedfs-db`
and stamps `helm.sh/resource-policy: keep` so the `seaweedfs-system`
upgrade (which no longer renders the Cluster) does not delete it during
the transition. Renumbered from the consolidated branch's slot 40 to fit
fresh main (last migration is 41). `targetVersion` bumped 42 → 43.

The new HR's `upgrade.force: true` was dropped to match #2602's
platform-wide cleanup (Helm v4 in helm-controller v1.5.0 rejects
combining `--force-replace` with SSA's `--force-conflicts`).

## Verification

- `helm template` renders `seaweedfs-db` HR cleanly with
`healthCheckExprs`, `waitStrategy: poller`, and `upgrade: { remediation:
{ retries: -1 } }` (no `force: true`).
- Migration 42 syntax-checks under `sh -n`.
- All four PR commits GPG-signed + DCO-signed-off.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* SeaweedFS database is now available as a separate, independently
configurable component with customizable replicas, storage size, and
storage class options.

* **Chores**
* Restructured database configuration from the main application into a
dedicated component for improved modularity and initialization order.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2601?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…(folds #2612) (#2602)

## What this PR does

Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and
the embedded management-cluster manifests, and folds in the chart fixes
that become hard errors under the new strict helm-controller v1.5.

Flux v2.8's helm-controller v1.5.0 ships:
- **Server-Side Apply with `--force-conflicts`** — strict CRD schema
enforcement; misplaced fields (silently dropped on v2.7) now fail the
apply.
- **kstatus-based health checking** by default — parent HR waits for
every applied resource (including child HRs) to be Ready before flipping
its own Ready, surfacing latent ordering deadlocks.
- **Helm v4 embedded** — `upgrade.force: true` is deprecated and now
conflicts with SSA.
- **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper
readiness gating (used in PR #2601 split).

Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same
Flux upgrade triggers it.

## Commits

**Flux upgrade itself:**
- `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts
to v0.48.0; web UI opt-in.
- `feat(flux)`: regenerate embedded management-cluster manifests via
`make update` in `packages/core/flux-aio` (timoni bundle build).

**Chart fixes for strict SSA — fields the chart sent that v2.7 silently
dropped, v2.8 rejects:**
- `fix(kubevirt-instancetypes)`: drop persistent strip that produced
null `preferredTPM` (folds #2612).
- `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`,
`minimumUptimeSecondsForBounce` from inside `automationOptions` to
direct children of `spec`.
- `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a
phantom `template.spec`.
- `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is
set.
- `fix(platform)`: drop deprecated `upgrade.force: true` from
HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths.

**Ordering / deadlock fixes under v2.8 kstatus:**
- `fix(vpa)`: break circular wait between parent install and nested
`vpa-for-vpa` HR.
- `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant
addon child HRs (parent waits on child via kstatus, child waited on
parent — deadlock).

**E2E waits for v2.8 kstatus timing:**
- `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's
faster dispatch).
- `test(e2e)`: wait for parent HR Ready before downstream asserts in
`run-kubernetes.sh` and `vminstance.bats`.

## Scope discipline

This PR is part of the split of #2619 (the consolidated CI fixes branch)
into review-friendly pieces. Companion PRs:
- **PR #2601** (seaweedfs split) — folded into this PR (commits
`29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the
seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption
migration 43 (targetVersion 44), and the configurable db resources all
land here, because the strict-SSA `upgrade.force` removal and the
kstatus parent-HR timeout bump only make sense together with the split.
#2601 is superseded.
- **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) —
independent, lands separately.
- Several smaller standalone fixes lifted out of #2619 (startup probes,
cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC
test improvements) — opened as separate PRs.

## Verification

- `helm template` renders cleanly for both `fluxcd` and
`fluxcd-operator` packages with `web.enabled=false` (default) and
`web.enabled=true`.
- Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0
/ v2.1.0 controller image strings.
- No references to the v0.39-removed `--disable-wait-interruption` flag
anywhere in `packages/` or `internal/`.

### Release note

```release-note
Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking).

When upgrading existing clusters:
- Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too.
- HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it.
- KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass.
- KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained.
- FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Flux Status web UI: optional web server, config secret, service port,
HTTPRoute/Ingress, network policy, and RBAC roles.
  * New SeaweedFS DB Helm chart and optional managed DB release.

* **Improvements**
* CRD/schema enhancements: new provider kinds, validations, variant
option, and external checksum refs.
* Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for
reliability.

* **Chores**
* Bumped Flux Operator and Flux versions; documentation links updated to
fluxoperator.dev.
* **Bug Fixes**
  * Removed aggressive HelmRelease force-upgrade/install flags.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Aligns kubevirt-csi-node HR with every other tenant addon HR in
packages/apps/kubernetes/templates/helmreleases/ (cilium, coredns,
ingress-nginx, etc.) — all set timeout: 10m, csi.yaml was the only
exception. 9f9d8f8 ("Allign timeouts for HelmReleases") missed it.

Symptom: e2e NFS test (hack/e2e-apps/run-kubernetes.sh) times out
waiting for nfs-test-pod to reach Succeeded. Cozyreport events show
the tenant csi HR running Install -> Uninstall -> Install remediation
cycles. With install.remediation.retries: -1 and the default 5m Helm
timeout, kstatus marks the kubevirt-csi-node DaemonSet 'InProgress'
past 5m on a slow tenant control-plane bringup (Kamaji APIs flapping,
VM image pulls), helm-controller times out, the remediation uninstalls
the DaemonSet, then reinstalls. Any volume mount in flight at that
moment is wedged.

Observed on PR #2619 CI run 26142986969: csi HR remediation at
~07:07 happened ~10 min into the nfs-test-pod wait, killing the
mount and timing out the pod-Succeeded condition.

10m matches the surrounding HRs and gives the DaemonSet headroom
without changing remediation semantics on truly-broken installs.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 866588f)

Aligns kubevirt-csi-node HR with every other tenant addon HR in
packages/apps/kubernetes/templates/helmreleases/ (cilium, coredns,
ingress-nginx, etc.) — all set timeout: 10m, csi.yaml was the only
exception. 9f9d8f8 ("Allign timeouts for HelmReleases") missed it.

Symptom: e2e NFS test (hack/e2e-apps/run-kubernetes.sh) times out
waiting for nfs-test-pod to reach Succeeded. Cozyreport events show
the tenant csi HR running Install -> Uninstall -> Install remediation
cycles. With install.remediation.retries: -1 and the default 5m Helm
timeout, kstatus marks the kubevirt-csi-node DaemonSet 'InProgress'
past 5m on a slow tenant control-plane bringup (Kamaji APIs flapping,
VM image pulls), helm-controller times out, the remediation uninstalls
the DaemonSet, then reinstalls. Any volume mount in flight at that
moment is wedged.

Observed on PR #2619 CI run 26142986969: csi HR remediation at
~07:07 happened ~10 min into the nfs-test-pod wait, killing the
mount and timing out the pod-Succeeded condition.

10m matches the surrounding HRs and gives the DaemonSet headroom
without changing remediation semantics on truly-broken installs.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 866588f)

…ay.bats tenant teardown (#2558)

## What this PR does

Drops the 3× retry loop on `Run E2E tests` and `Install Cozystack into
sandbox`. `Prepare environment` keeps its 3× retry — that step is pure
infrastructure (Talos image download, sandbox VM boot, network) where
transient runner hiccups warrant a retry.

On failure, the test step now captures `kubectl get hr -A -o wide` and
`kubectl get events -A` under a collapsible group so triage starts with
the actual broken-state snapshot.

> [!NOTE]
> An earlier revision of this PR also doubled every bats timeout. That
commit was dropped in a rebase and is intentionally **not restored**:
the timeout class that actually matters (per-app HR-Ready waits) has
since been standardized at 5m on `main` (7b9f286), making a blanket 2×
redundant.

**Fixes gateway.bats teardown leakage.** The nested-tenant tests deleted
tenants fire-and-forget, parent and child back-to-back. The leftover
uninstalls (each blocked on a cleanup Job, parents wedged on
still-terminating child namespaces) plus one mid-install child HR
occupied exactly 5 workers on the `--concurrent=5` tenants
helm-controller shard, starving whichever app test ran next — observed
as the harbor HR sitting unreconciled for its whole 5m HR-Ready budget
in [run
27020081550](https://github.com/cozystack/cozystack/actions/runs/27020081550),
surfaced by this PR's own retry removal + diagnostics dump. Teardown now
deletes child→parent with hard `wait hr --for=delete` between, so a
wedged tenant uninstall fails gateway.bats itself, not an innocent
neighbor.

## Why

Audit of 30 successful PR runs found that across 5 sampled failure
attempts, **25/25 retries** on `Run E2E tests` failed — the retry loop
never recovered a flake, only stretched deterministic failures and
tripled diagnostic wall-time. Same data shape on `Install Cozystack`.

Beyond wasted CI time, the retry was hiding ~10 deterministic bugs (Helm
namespace-ownership conflict, seaweedfs HR timeout, harbor BucketInfo
wiring, vminstance disk race, etc.). Each failure looked like a "flake"
because the retry sometimes coincided with whatever transient state had
cleared — the retry never fixed the bug, just delayed surfacing.

## Dependencies

The deterministic bugs the retry was masking are now fixed on `main`:
- ✅ **#2508** — installer namespace bootstrap (Helm namespace-ownership
conflict on cold install) — merged
- ✅ **#2509** — operator HelmRelease config knobs (`seaweedfs-system`
2-min wait race within Flux's 5-min reconcile windows) — merged
- ✅ **#2528** — harbor bucket-secret + BucketInfo gating (harbor
`ValuesError` on first install) — merged
- ✅ **#2529** — objectstorage-controller BucketAccess conflict retry —
merged

Companion PRs in the #2619 split (independent of this PR,
ordering-wise):
- **#2602** — Flux v2.8.0 + chart fixes
- **#2601** — seaweedfs-system split

This PR does NOT depend on #2602/#2601 — it now touches only the
workflow file and gateway.bats teardown, both on top of fresh `main`.

Surfaced from #2500.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* CI prepare-environment step now reports plain attempt counts with
clear success/failure messages.
* Install and per-app test steps no longer retry; each runs once and
fails immediately on error. Failed apps log diagnostics and job proceeds
to remaining apps while overall job fails.
* **Tests**
* End-to-end tests and install/prepare flows use longer, more tolerant
timeouts and added existence polling to reduce flakiness and improve
diagnostics.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2558?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Aligns kubevirt-csi-node HR with every other tenant addon HR in
packages/apps/kubernetes/templates/helmreleases/ (cilium, coredns,
ingress-nginx, etc.) — all set timeout: 10m, csi.yaml was the only
exception. 9f9d8f8 ("Allign timeouts for HelmReleases") missed it.

Symptom: e2e NFS test (hack/e2e-apps/run-kubernetes.sh) times out
waiting for nfs-test-pod to reach Succeeded. Cozyreport events show
the tenant csi HR running Install -> Uninstall -> Install remediation
cycles. With install.remediation.retries: -1 and the default 5m Helm
timeout, kstatus marks the kubevirt-csi-node DaemonSet 'InProgress'
past 5m on a slow tenant control-plane bringup (Kamaji APIs flapping,
VM image pulls), helm-controller times out, the remediation uninstalls
the DaemonSet, then reinstalls. Any volume mount in flight at that
moment is wedged.

Observed on PR #2619 CI run 26142986969: csi HR remediation at
~07:07 happened ~10 min into the nfs-test-pod wait, killing the
mount and timing out the pod-Succeeded condition.

10m matches the surrounding HRs and gives the DaemonSet headroom
without changing remediation semantics on truly-broken installs.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 866588f)

Aligns kubevirt-csi-node HR with every other tenant addon HR in
packages/apps/kubernetes/templates/helmreleases/ (cilium, coredns,
ingress-nginx, etc.) — all set timeout: 10m, csi.yaml was the only
exception. 9f9d8f8 ("Allign timeouts for HelmReleases") missed it.

Symptom: e2e NFS test (hack/e2e-apps/run-kubernetes.sh) times out
waiting for nfs-test-pod to reach Succeeded. Cozyreport events show
the tenant csi HR running Install -> Uninstall -> Install remediation
cycles. With install.remediation.retries: -1 and the default 5m Helm
timeout, kstatus marks the kubevirt-csi-node DaemonSet 'InProgress'
past 5m on a slow tenant control-plane bringup (Kamaji APIs flapping,
VM image pulls), helm-controller times out, the remediation uninstalls
the DaemonSet, then reinstalls. Any volume mount in flight at that
moment is wedged.

Observed on PR #2619 CI run 26142986969: csi HR remediation at
~07:07 happened ~10 min into the nfs-test-pod wait, killing the
mount and timing out the pod-Succeeded condition.

10m matches the surrounding HRs and gives the DaemonSet headroom
without changing remediation semantics on truly-broken installs.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 866588f)

## What this PR does

Adds `spec.timeout: 10m` to the `kubevirt-csi-node` HelmRelease template
in `packages/apps/kubernetes/templates/helmreleases/csi.yaml`, aligning
it with every other tenant addon HR in the same directory (cilium,
coredns, ingress-nginx, etc. — all set `timeout: 10m`). `csi.yaml` was
the only one missing it; commit 9f9d8f8 ("Align timeouts for
HelmReleases") missed this template.

## Symptom

E2E NFS test times out waiting for `nfs-test-pod` to reach `Succeeded`.
Cozyreport events show the tenant `csi` HR running an Install →
Uninstall → Install remediation cycle. With
`install.remediation.retries: -1` and the default 5m Helm timeout,
kstatus marks the `kubevirt-csi-node` DaemonSet `InProgress` past 5m on
a slow tenant control-plane bringup (Kamaji APIs flapping, VM image
pulls), helm-controller times out, the remediation uninstalls the
DaemonSet, then reinstalls. Any volume mount in flight at that moment is
wedged.

## Origin

Single commit lifted unchanged from #2619.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated the Container Storage Interface (CSI) Helm release settings to
include a `spec.timeout` of **10 minutes** when the etcd namespace value
is set (keeping the existing `spec.interval` at **5 minutes**).
<!-- end of auto-generated comment: release notes by coderabbit.ai -->