test(e2e): give tenant workers two vCPUs - #3839
Conversation
The tenant worker boots Talos and unpacks the kubelet image on a single vCPU whose CFS quota also covers the QEMU emulator and IO threads, and measurements on both quiet and loaded runs show the compute container pinned at 93-98% of that one-core ceiling for the whole node-Ready budget. The join deadline is then decided entirely by how much CPU work the boot happens to need, and under parallel e2e load it does not fit. Doubling the worker to u1.large raises the ceiling instead of the weight, which measurements showed changes nothing since the workers never lose a scheduling fight. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughThe Kubernetes end-to-end test setup changes the tenant worker node group instance type from ChangesTenant worker sizing
Estimated code review effort: 1 (Trivial) | ~2 minutes Merge Risk: 🔵 Low · up to Tenant workers will use the two-vCPU u1.large profile, but the target e2e cluster must provide that profile or the suite will fail before executing tests; merge is reasonable with explicit owner confirmation. Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
## What this PR does Follow-up to #3839, which moved the e2e tenant workers to `u1.large` to give them a second vCPU. That instancetype doubles memory along with the vCPUs, and four 8Gi workers do not fit on the sandbox beside the rest of the suite. On the first runs after it landed, the second worker of each cluster stayed Pending with `0/3 nodes are available: 3 Insufficient memory`, so the node group never reached two Ready nodes and node-join failed for a reason unrelated to the CPU ceiling the change was meant to test. The memory estimate in that PR body counted the workers against total node memory and did not account for what the rest of the suite already occupies. This sizes the group with `resources.cpu` and `resources.memory` instead, which keeps the two vCPUs at the 4Gi the workers had before. The chart omits the instancetype from the VM whenever a group sets both fields, since KubeVirt rejects a VM that references an instancetype and also overrides CPU or memory, and the values schema still lists `instanceType` as required, so it stays in the values as the fallback it now is. The measurement from those runs is worth keeping even though the runs were spoiled. Where the workers did start, the two-scrape capture read 1.46 to 1.93 against the new ceiling of 2.0, with 2 to 5 percent of CFS periods throttled, against 0.93 to 0.98 of a 1.0 ceiling at 8 to 13 percent throttled before. The ceiling was binding, and raising it does loosen it; whether that is enough to make the join deadline is what the next runs measure. ### Screenshots Not applicable, no UI change. ### Downstream repositories The diff is one values block inside `hack/e2e-chainsaw/_lib/run-kubernetes.sh`. Nothing under `hack/` is moved or renamed, no make target changes behaviour, no package, schema, default or CRD changes. Nothing in the trigger map is touched. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): tenant workers in the kubernetes e2e suites keep their two vCPUs but go back to 4Gi of memory, since the larger instancetype left workers unschedulable on the sandbox ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Configuration** * Updated tenant worker nodes to use `u1.medium` instances. * Set each worker’s resources to 2 CPUs and 4 GiB of memory. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
What this PR does
One line in the kubernetes e2e suite values: tenant workers move from
u1.mediumtou1.large, one vCPU to two.The reason is measured, in two independent ways on #3513. A single-scrape reading on a quiet branch run and the two-scrape instrument from #3836 on a loaded run with six parallel e2e workflows agree: the worker's compute container sits at 93 to 98 percent of its one-core ceiling for the whole node-Ready budget, the sandbox nodes under it show ~0.05 percent steal, and raising the CPU request changes nothing because the workers never lose a scheduling fight. The binding constraint is capacity: one vCPU has to boot Talos and unpack the kubelet image, and its quota also covers the QEMU emulator and IO threads.
This is an experiment as much as a fix: the suites run the same assertions, and the next waves of runs under parallel load measure whether the join flake documented in #3513 disappears with the ceiling doubled. The diagnostics landed in #3804 and #3836 stay aboard, so the same instrument reports the new profile.
Memory math on the sandbox: two suites with two workers each move from 4Gi to 8Gi guests, and the virt-launcher pods spread across three 24Gi nodes, so the placement still fits.
Screenshots
Not applicable, no UI change.
Downstream repositories
The diff is one value inside
hack/e2e-chainsaw/_lib/run-kubernetes.sh. Nothing underhack/is moved or renamed, no make target changes behaviour, no package, schema, default or CRD changes. Nothing in the trigger map is touched.Release note
Summary by CodeRabbit