fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) by kvaps · Pull Request #2957 · cozystack/cozystack

Andrei Kvapil (kvaps) · 2026-06-19T21:02:55Z

What

The e2e install pipeline still flakes on a loaded runner, failing at a different point each run. Root-caused via repeated runs and a live SSH breakpoint; the platform itself is fine (the components do converge), the failures are slowness + self-inflicted churn under resource pressure.

Fixes

VPA updater OOM — the vertical-pod-autoscaler-updater memory limit was 110Mi. The updater watches every pod in the cluster, so on a full install it OOM-kills → ProgressDeadlineExceeded → Flux fails the HelmRelease "early due to stalled resources", cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raised to 512Mi (request 200Mi).
VPA-for-VPA eviction churn — the two VPA-for-VPA VerticalPodAutoscaler objects ran updateMode: Auto, evicting the VPA recommender(s) during install and adding pod churn on an already busy cluster. Switched to Initial, matching the existing monitoring/etcd VPA->Initial fixes.
Install wait budgets — components reach readiness but overshoot tight waits on a busy runner: keycloak HR chain 10m→20m, tenant aggregate wait 10m→20m, per-app tenant readiness waits 5m→10m.

(The crust-gather failure-snapshot timeout that kept whole jobs hanging to the 3h cancel is already on main.)

Release note

NONE

Summary by CodeRabbit

Bug Fixes
- Increased several installation and readiness wait times to better handle slower environment startup and reduce premature failures.
- Updated vertical autoscaling behavior to avoid unnecessary pod restarts during deployment.
- Raised resource limits for the autoscaling updater to improve reliability under load.
Tests
- Aligned automated checks with the new autoscaling behavior and longer readiness expectations.

coderabbitai · 2026-06-19T21:03:03Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 94ce1013-4c94-4e08-a94c-d45592b597ff

📥 Commits

Reviewing files that changed from the base of the PR and between 20e4eab and d6f66d2.

📒 Files selected for processing (4)

hack/e2e-install-cozystack.bats
packages/system/vertical-pod-autoscaler/templates/vpa-for-vpa.yaml
packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml
packages/system/vertical-pod-autoscaler/values.yaml

📝 Walkthrough

Walkthrough

This PR extends e2e test wait timeouts for tenant, service, and Keycloak readiness checks in hack/e2e-install-cozystack.bats, and changes VerticalPodAutoscaler updateMode from Recreate to Initial in vpa-for-vpa.yaml, updating corresponding tests and increasing updater container memory limits/requests in values.yaml.

Changes

E2E Readiness Timeout Increases

Layer / File(s)	Summary
Tenant and service readiness waits `hack/e2e-install-cozystack.bats`	Extends hr/tenant-root wait to 20m and increases downstream waits (ingress, etcd, VictoriaMetrics, VM clusters, Grafana) from 5m to 10m.
Keycloak stack readiness wait `hack/e2e-install-cozystack.bats`	Increases Keycloak-related HelmRelease wait timeout from 10m to 20m.

VPA Update Mode and Resource Sizing

Layer / File(s)	Summary
Template updateMode change `packages/system/vertical-pod-autoscaler/templates/vpa-for-vpa.yaml`	Changes updatePolicy.updateMode from Recreate to Initial for both VerticalPodAutoscaler resources, with explanatory comments.
Test assertions update `packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml`	Updates test case description and assertion to expect updateMode Initial instead of Recreate.
Updater memory sizing `packages/system/vertical-pod-autoscaler/values.yaml`	Increases updater memory limit (110Mi→512Mi) and request (110Mi→200Mi) with rationale comments.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

cozystack/cozystack#1359: Both PRs modify the same "Configure Tenant and wait for applications" readiness logic/timeouts in hack/e2e-install-cozystack.bats.
cozystack/cozystack#2558: Both PRs change e2e readiness/wait timeouts for tenant/app and Keycloak OIDC stack HelmRelease waits in the same test file.
cozystack/cozystack#3036: Both PRs widen etcd-related e2e timeout budgets, though in different bats files.

Suggested labels

kind/flake, area/kubernetes

Suggested reviewers

lllamnyp
androndo
IvanHunters
sircthulhu
myasnikovdaniil

A rabbit hops with patience new,
Twenty minutes, not just a few. 🐇
VPA settles in, gentle and slow,
Initial sizing, no eviction woe.
More memory room to breathe and grow—
Timeouts widened, tests still glow!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: fixing e2e install flakiness via VPA adjustments and longer wait budgets.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/e2e-green-on-ci-next

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

…tabilization External-mode LinstorCluster driving linstor-csi at the blockstor apiserver; blockstor controller/apiserver/satellite in cozy-linstor. Pins the satellite zvol-device-wait and apiserver RD-404 fix images for e2e validation, and carries the VPA updater OOM/churn fix (#2957) so heavy JVM apps stop OOMKilling on the constrained e2e stand. Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>

The e2e install still flakes on a busy runner where components reach readiness but overshoot tight wait budgets: - keycloak HR chain (keycloak-db CNPG -> keycloak StatefulSet -> keycloak-configure) reaches Ready ~15-18m in: 10m -> 20m. - tenant aggregate wait (hr/etcd hr/ingress hr/monitoring hr/seaweedfs hr/tenant-root): the tenant stack only starts creating pods ~9-10m in and then needs several more minutes: 10m -> 20m. - per-app tenant readiness waits (ingress, etcd, VictoriaLogs/Metrics clusters, grafana-db, grafana): 5m -> 10m. Diagnosed via a live SSH breakpoint: the HRs all do reach Ready, just outside the old budgets. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>

…tion churn Two root causes of e2e install flakiness, both in the VPA package: 1. The updater memory limit was 110Mi. The updater watches every pod in the cluster, so on a full cozystack install it OOM-kills, which trips the Deployment's ProgressDeadlineExceeded and makes Flux fail the HelmRelease 'early due to stalled resources' - cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raise the limit to 512Mi (request 200Mi); the recommender is already unlimited. 2. The two VPA-for-VPA VerticalPodAutoscaler objects ran updateMode: Auto, which evicts the VPA recommender(s) during install and adds pod churn on an already busy cluster. Switch to Initial, matching the existing monitoring/etcd VPA->Initial fixes. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> Signed-off-by: Aleksei Sviridkin <f@lex.la>

Aleksei Sviridkin (lexfrei) · 2026-07-01T00:30:22Z

Rebased onto current main to revive this — it had gone stale and the branch was showing conflicts. Your fix is unchanged: updater OOM headroom, VPA-for-VPA Initial, and the widened install wait budgets.

Two merge-driven notes: main had already migrated the VPA-for-VPA updateMode from the deprecated Auto to Recreate, so the effective delta here is now Recreate → Initial; and I brought the updateMode helm-unittest fixture and the tenant-root wait comment in line with the 20m budget. Left as draft — yours to take from here.

gemini-code-assist · 2026-07-01T01:36:32Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses flakiness in the E2E installation pipeline caused by resource pressure and tight timing constraints. By optimizing resource limits for the VPA updater, adjusting VPA eviction policies to reduce churn, and increasing wait budgets for critical components, the changes ensure more stable and reliable deployments on busy runners.

Highlights

VPA Updater Resource Allocation: Increased the memory limits and requests for the vertical-pod-autoscaler-updater to prevent OOM-kills during cluster installation.
VPA Eviction Policy: Switched the VPA-for-VPA update mode from 'Recreate' to 'Initial' to eliminate unnecessary pod churn and recommender evictions during the installation process.
Installation Wait Budgets: Extended various timeout thresholds in the E2E installation pipeline to accommodate resource-constrained runners and prevent premature failures.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	Gemini (@gemini-code-assist)	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request increases timeouts in the E2E installation script to prevent premature expiration on loaded runners. It also changes the update mode of the vertical-pod-autoscaler (VPA) components from 'Recreate' to 'Initial' to reduce pod churn during installation, and increases the memory limits and requests for the VPA updater to prevent OOM-kills on full installations. I have no feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Aleksei Sviridkin (lexfrei)

Rebased onto current main to revive this install-pipeline flaking fix (VPA updater OOM headroom + wait-budget tuning). Cross-checked with Codex and two independent review passes — no blockers; e2e green and all checks pass. LGTM.

github-actions Bot added the size/S This PR changes 10-29 lines, ignoring generated files label Jun 19, 2026

github-actions Bot added the area/testing Issues or PRs related to testing (e2e, bats, unit tests) label Jun 19, 2026

Andrei Kvapil (kvaps) added the debug Debugging in progress label Jun 19, 2026

github-actions Bot added size/M This PR changes 30-99 lines, ignoring generated files and removed size/S This PR changes 10-29 lines, ignoring generated files labels Jun 20, 2026

Base automatically changed from ci-next to main June 22, 2026 04:19

Andrei Kvapil (kvaps) force-pushed the fix/e2e-green-on-ci-next branch from 3e2919a to 17568d6 Compare June 22, 2026 10:05

Andrei Kvapil (kvaps) changed the title ~~test(e2e): drive ci-next stabilization to a green e2e run~~ fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) Jun 22, 2026

github-actions Bot added the kind/bug Categorizes issue or PR as related to a bug label Jun 22, 2026

Andrei Kvapil (kvaps) mentioned this pull request Jun 24, 2026

PoC: replace LINSTOR with Blockstor (external-mode LinstorCluster + blockstor controller/apiserver/satellite) #2942

Open

Andrei Kvapil (kvaps) and others added 2 commits July 1, 2026 03:27

Aleksei Sviridkin (lexfrei) force-pushed the fix/e2e-green-on-ci-next branch from 17568d6 to d6f66d2 Compare July 1, 2026 00:29

Aleksei Sviridkin (lexfrei) marked this pull request as ready for review July 1, 2026 01:36

Aleksei Sviridkin (lexfrei) requested review from IvanHunters, Andrey Kolkov (androndo), Aleksei Sviridkin (lexfrei), Timofei Larkin (lllamnyp), myasnikovdaniil and Kirill Ilin (sircthulhu) as code owners July 1, 2026 01:36

gemini-code-assist Bot reviewed Jul 1, 2026

View reviewed changes

Aleksei Sviridkin (lexfrei) approved these changes Jul 1, 2026

View reviewed changes

Aleksei Sviridkin (lexfrei) merged commit 9c18260 into main Jul 1, 2026
22 of 23 checks passed

Aleksei Sviridkin (lexfrei) deleted the fix/e2e-green-on-ci-next branch July 1, 2026 03:00

This was referenced Jul 13, 2026

test(e2e): add chainsaw-native release upgrade testing lane #3276

Open

fix(platform): route OIDC backend calls through the in-cluster Keycloak by default #3289

Merged

coderabbitai Bot mentioned this pull request Jul 23, 2026

fix(e2e): keep failure snapshots complete and gate tenant patch on aggregated-API auth #3444

Merged

11 tasks

Navigation Menu

Uh oh!

fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) - #2957

fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets)#2957
Aleksei Sviridkin (lexfrei) merged 2 commits into
mainfrom
fix/e2e-green-on-ci-next

Andrei Kvapil (kvaps) commented Jun 19, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jun 19, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

Suggested reviewers

Uh oh!

Aleksei Sviridkin (lexfrei) commented Jul 1, 2026

Uh oh!

gemini-code-assist Bot commented Jul 1, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Aleksei Sviridkin (lexfrei) left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Andrei Kvapil (kvaps) commented Jun 19, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Fixes

Release note

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested labels

Suggested reviewers

Uh oh!

Aleksei Sviridkin (lexfrei) commented Jul 1, 2026

Uh oh!

gemini-code-assist Bot commented Jul 1, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Aleksei Sviridkin (lexfrei) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Andrei Kvapil (kvaps) commented Jun 19, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 19, 2026 •

edited

Loading