ci: run the slowest test packages first - #29457
Draft
ibetitsmike wants to merge 6 commits into
Draft
ibetitsmike wants to merge 6 commits into
ibetitsmike wants to merge 6 commits into
Conversation
go test starts packages in argument order, so ./... (alphabetical) left the heaviest packages for the end of the run: in the test-go-pg ubuntu job enterprise/coderd (110s) did not start until t=313s of a 423s run. scripts/test_packages.sh expands the pattern list with the slowest packages first so the -p scheduler packs the long tail behind them.
With every heavy package first, the light packages and their builds formed a 60s tail after the last heavy package finished. Start the five 100s+ packages, let the light packages fill the remaining slots, and run the medium packages once heavy slots free up.
…t count Three TEST_COUNT=1 runs on this branch: every heavy package first 356s, five heavy then light then medium 364s, ten heavy first 373s, against 364-423s for the last six main runs. The differences are within noise; the five-first order keeps coderd, the future critical path, least inflated by contention.
This was referenced Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
go teststarts packages in argument order, and./...expands alphabetically, so the heaviest packages started last and set the end of the run. In the baselinetest-go-pg (ubuntu-latest)job (run 35182300886, 423s test step)enterprise/coderd(110s) did not start until t=313s.scripts/test_packages.shexpands theTEST_PACKAGESpatterns withgo listand prints the five 100s+ packages first, then the light packages (their builds and runs fill the remaining slots instead of forming a tail), then the 20-100s packages.make test,make test-raceandmake test-timingspass the ordered list to gotestsum. No test code changes; the CI action is unchanged because it already goes throughmake test. The static list only needs to be roughly right: packages missing from the pattern set are skipped and everything else keepsgo listorder (verified with the macOS/Windows pattern./cli/... ./enterprise/cli/... ./agent/...).Ordering is confirmed locally with
go test -json -p 1 ./scripts/release ./agent/agentchat ./tailnet/proto: packages start in argument order.Before / After
PR CI normally runs with the Go test cache (only main sets
TEST_COUNT=1), so the numbers below come from three runs on this branch withtest-count: "1"temporarily forced (that override is reverted in the final commit):DONE 31432 testswallThe three orderings are within noise of each other and about 10% under main. The job is close to saturated: starting eight heavy packages at once inflates each of them (coderd 224s to 271s), which is why the five-first order was kept: it leaves coderd, the critical path once the fixture PRs land, least inflated (238s). Further wall-time gains come from the fixture and clock PRs in this series, not from ordering.
Validation
shellcheck scripts/test_packages.sh./scripts/test_packages.shlists the same 373 packages asgo list ./...; the macOS/Windows pattern yields 45 withcliandenterprise/clifirstmake test TEST_PACKAGES='./coderd/httpapi/... ./scripts/release' TEST_COUNT=1passestest-go-pgpassed on ubuntu, macOS (BSD awk needed newline-free-vvalues, fixed), Windows and PG 17 on the measurement commitsmake pre-commit, since the Makefile changed) passedPart of the test-suite speedup series: #29458 (mock clocks), #29459 (notifications golden), #29460 (migrations chain), #29461 (dead code), #29462 (server tailnet DERP poll fix). All independent.
Remote validation
Remote run on a dogfood workspace (separate compute), chat https://dogfood.cdr.dev/agents/5bbe73d0-866a-4841-9cc3-c5e5b4d97e44, tested head
ee5d3be95da(and the earlier37754184d0f):./scripts/test_packages.shlists 373 packages, identical set togo list ./..., no duplicates, five heavy packages first;make test TEST_PACKAGES='./coderd/httpapi/... ./scripts/release' TEST_COUNT=1ran exactly those two packages. Endorsed verdict: PASS.