test(e2e): widen etcd convergence budget to 5m in Talos bootstrap - #3036
Conversation
The "Bootstrap Talos cluster" step times out the etcd-members health gate at 180s. A 3-node etcd converges via Talos's serialized learner promotion (one learner admitted at a time), so on a loaded ephemeral runner this deterministic path can approach ~3m wall clock and photo-finish against the 180s ceiling — failing a cluster that is in fact healthy. The wait is already event-driven; only the ceiling was too tight. Widen it to 300s. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe Talos cluster bootstrap script now waits longer for etcd readiness before failing, and its inline comment was updated. The later logging and exit behavior after the check remain unchanged. ChangesTalos cluster bootstrap wait
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~5 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses intermittent E2E test failures occurring during the Talos cluster bootstrap phase. By extending the timeout for etcd member convergence, the change accommodates the natural, serialized promotion process of etcd members, which can occasionally exceed the previous 3-minute limit under high system load without indicating an actual cluster health issue. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request increases the timeout for waiting until etcd is healthy from 180 to 300 seconds in hack/e2e-prepare-cluster.bats to prevent premature timeouts on loaded ephemeral runners. The review feedback notes that the PR title uses an invalid scope (e2e instead of tests or hack) and still contains a placeholder release note, violating the repository's commit and PR guidelines.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| # Wait until etcd is healthy. | ||
| # Budget is 5m, not 3m: a 3-node etcd converges via Talos's serialized |
There was a problem hiding this comment.
The pull request does not fully comply with the repository's commit and PR requirements:
- Invalid Scope in PR Title/Commit: The PR title uses the scope
e2e(test(e2e): ...). According to the Repository Style Guide (lines 40-46),e2eis not a valid scope. Please update the scope to a valid one, such astestsorhack(e.g.,test(tests): ...ortest(hack): ...). - Placeholder Release Note: The release-note block in the PR body is still using the default placeholder template (
type(scope): human-readable changelog entry). Please update it with a proper human-readable changelog entry describing this change.
References
- Commit and PR Requirements: Each commit must follow Conventional Commits format with a valid scope (e.g.,
testsorhackinstead ofe2e). The PR body must contain a completed release-note block rather than the default placeholder. (link)
IvanHunters
left a comment
There was a problem hiding this comment.
Reviewed the diff. Timeout-only change in hack/e2e-prepare-cluster.bats, no prod path touched; 5m budget aligned with existing HelmRelease/install timeouts. LGTM.
Problem
The
Bootstrap Talos clusterE2E step fails intermittently with❌ Test failed: Bootstrap Talos cluster (exit 1)even though the cluster is healthy. The exit comes from the etcd-members convergence gate, which is bounded attimeout 180.A 3-node etcd converges via Talos's serialized learner promotion — etcd admits one learner at a time, so the third member is only promoted after the second has fully caught up. On a loaded ephemeral runner this deterministic path can legitimately approach ~3 minutes of wall clock and photo-finish against the 180s ceiling, timing out a cluster that has in fact converged (every member reaches
Health check successful; no kernel panic / OOM / boot failure / network-unreachable in dmesg). Observed: a run where the third member reached etcd health at +178.6s against the 180s budget, failing ~1.4s short.Fix
Widen the etcd-members convergence budget from 180s to 300s. The wait is already event-driven (
until talosctl etcd members …; do sleep 1; done) — only the ceiling was too tight. No retry is added and the dmesg-on-failure diagnostic is preserved.300s matches the repo's other in-line budgets for deterministic convergence (HelmRelease-ready 5m, harbor/NFS 10m, install 15m).
Why not just rerun
This is not environmental noise: the cluster converges correctly every time, it is only timed out. Rerunning wastes a full ~1h E2E cycle to re-roll the dice on whether the runner shaves the last ~2s. Correcting the mistuned budget removes the photo-finish.
Summary by CodeRabbit