Skip to content

test(e2e): give tenant workers two vCPUs - #3839

Merged
Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
test/e2e-worker-two-vcpu
Aug 15, 2026
Merged

test(e2e): give tenant workers two vCPUs#3839
Aleksei Sviridkin (lexfrei) merged 1 commit into
mainfrom
test/e2e-worker-two-vcpu

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

What this PR does

One line in the kubernetes e2e suite values: tenant workers move from u1.medium to u1.large, one vCPU to two.

The reason is measured, in two independent ways on #3513. A single-scrape reading on a quiet branch run and the two-scrape instrument from #3836 on a loaded run with six parallel e2e workflows agree: the worker's compute container sits at 93 to 98 percent of its one-core ceiling for the whole node-Ready budget, the sandbox nodes under it show ~0.05 percent steal, and raising the CPU request changes nothing because the workers never lose a scheduling fight. The binding constraint is capacity: one vCPU has to boot Talos and unpack the kubelet image, and its quota also covers the QEMU emulator and IO threads.

This is an experiment as much as a fix: the suites run the same assertions, and the next waves of runs under parallel load measure whether the join flake documented in #3513 disappears with the ceiling doubled. The diagnostics landed in #3804 and #3836 stay aboard, so the same instrument reports the new profile.

Memory math on the sandbox: two suites with two workers each move from 4Gi to 8Gi guests, and the virt-launcher pods spread across three 24Gi nodes, so the placement still fits.

Screenshots

Not applicable, no UI change.

Downstream repositories

The diff is one value inside hack/e2e-chainsaw/_lib/run-kubernetes.sh. Nothing under hack/ is moved or renamed, no make target changes behaviour, no package, schema, default or CRD changes. Nothing in the trigger map is touched.

Release note

test(e2e): tenant workers in the kubernetes e2e suites run with two vCPUs, since measurements showed the single-vCPU worker saturating its CPU ceiling for the entire node-join budget

Summary by CodeRabbit

  • Chores
    • Updated the tenant worker node configuration to use a larger instance type, improving available capacity for end-to-end Kubernetes testing.

The tenant worker boots Talos and unpacks the kubelet image on a single
vCPU whose CFS quota also covers the QEMU emulator and IO threads, and
measurements on both quiet and loaded runs show the compute container
pinned at 93-98% of that one-core ceiling for the whole node-Ready
budget. The join deadline is then decided entirely by how much CPU work
the boot happens to need, and under parallel e2e load it does not fit.

Doubling the worker to u1.large raises the ceiling instead of the
weight, which measurements showed changes nothing since the workers
never lose a scheduling fight.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
@github-actions github-actions Bot added area/testing Issues or PRs related to testing (e2e, bats, unit tests) size/XS This PR changes 0-9 lines, ignoring generated files labels Aug 15, 2026
@coderabbitai

coderabbitai Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d55f1ba7-938e-4b63-9788-acb1f89264d3

📥 Commits

Reviewing files that changed from the base of the PR and between ef96292 and 3b49be0.

📒 Files selected for processing (1)
  • hack/e2e-chainsaw/_lib/run-kubernetes.sh

📝 Walkthrough

Walkthrough

The Kubernetes end-to-end test setup changes the tenant worker node group instance type from u1.medium to u1.large.

Changes

Tenant worker sizing

Layer / File(s) Summary
Update tenant worker instance type
hack/e2e-chainsaw/_lib/run-kubernetes.sh
The tenant worker node group now uses u1.large instead of u1.medium.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: 🔵 Low · up to 3b49b

Tenant workers will use the two-vCPU u1.large profile, but the target e2e cluster must provide that profile or the suite will fail before executing tests; merge is reasonable with explicit owner confirmation.

Suggested reviewers: kvaps

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main change: increasing tenant worker capacity to two vCPUs for e2e tests.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/e2e-worker-two-vcpu

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@lexfrei
Aleksei Sviridkin (lexfrei) merged commit 71c0fc1 into main Aug 15, 2026
14 of 15 checks passed
@lexfrei
Aleksei Sviridkin (lexfrei) deleted the test/e2e-worker-two-vcpu branch August 15, 2026 12:23
Aleksei Sviridkin (lexfrei) added a commit that referenced this pull request Aug 15, 2026
## What this PR does

Follow-up to #3839, which moved the e2e tenant workers to `u1.large` to
give them a second vCPU. That instancetype doubles memory along with the
vCPUs, and four 8Gi workers do not fit on the sandbox beside the rest of
the suite. On the first runs after it landed, the second worker of each
cluster stayed Pending with `0/3 nodes are available: 3 Insufficient
memory`, so the node group never reached two Ready nodes and node-join
failed for a reason unrelated to the CPU ceiling the change was meant to
test. The memory estimate in that PR body counted the workers against
total node memory and did not account for what the rest of the suite
already occupies.

This sizes the group with `resources.cpu` and `resources.memory`
instead, which keeps the two vCPUs at the 4Gi the workers had before.
The chart omits the instancetype from the VM whenever a group sets both
fields, since KubeVirt rejects a VM that references an instancetype and
also overrides CPU or memory, and the values schema still lists
`instanceType` as required, so it stays in the values as the fallback it
now is.

The measurement from those runs is worth keeping even though the runs
were spoiled. Where the workers did start, the two-scrape capture read
1.46 to 1.93 against the new ceiling of 2.0, with 2 to 5 percent of CFS
periods throttled, against 0.93 to 0.98 of a 1.0 ceiling at 8 to 13
percent throttled before. The ceiling was binding, and raising it does
loosen it; whether that is enough to make the join deadline is what the
next runs measure.

### Screenshots

Not applicable, no UI change.

### Downstream repositories

The diff is one values block inside
`hack/e2e-chainsaw/_lib/run-kubernetes.sh`. Nothing under `hack/` is
moved or renamed, no make target changes behaviour, no package, schema,
default or CRD changes. Nothing in the trigger map is touched.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(e2e): tenant workers in the kubernetes e2e suites keep their two vCPUs but go back to 4Gi of memory, since the larger instancetype left workers unschedulable on the sandbox
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Configuration**
  * Updated tenant worker nodes to use `u1.medium` instances.
  * Set each worker’s resources to 2 CPUs and 4 GiB of memory.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/testing Issues or PRs related to testing (e2e, bats, unit tests) size/XS This PR changes 0-9 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant