Skip to content

ci: run the slowest test packages first - #29457

Draft
ibetitsmike wants to merge 6 commits into
mainfrom
mike/ci-test-package-order
Draft

ibetitsmike wants to merge 6 commits into
mainfrom
mike/ci-test-package-order

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

go test starts packages in argument order, and ./... expands alphabetically, so the heaviest packages started last and set the end of the run. In the baseline test-go-pg (ubuntu-latest) job (run 35182300886, 423s test step) enterprise/coderd (110s) did not start until t=313s.

scripts/test_packages.sh expands the TEST_PACKAGES patterns with go list and prints the five 100s+ packages first, then the light packages (their builds and runs fill the remaining slots instead of forming a tail), then the 20-100s packages. make test, make test-race and make test-timings pass the ordered list to gotestsum. No test code changes; the CI action is unchanged because it already goes through make test. The static list only needs to be roughly right: packages missing from the pattern set are skipped and everything else keeps go list order (verified with the macOS/Windows pattern ./cli/... ./enterprise/cli/... ./agent/...).

Ordering is confirmed locally with go test -json -p 1 ./scripts/release ./agent/agentchat ./tailnet/proto: packages start in argument order.

Before / After

PR CI normally runs with the Go test cache (only main sets TEST_COUNT=1), so the numbers below come from three runs on this branch with test-count: "1" temporarily forced (that override is reverted in the final commit):

Ordering DONE 31432 tests wall
main, last six runs (alphabetical) 364s to 423s (mean 403s)
every heavy package first 356s
five heavy, then light, then medium (this PR) 364s
ten heavy first 373s

The three orderings are within noise of each other and about 10% under main. The job is close to saturated: starting eight heavy packages at once inflates each of them (coderd 224s to 271s), which is why the five-first order was kept: it leaves coderd, the critical path once the fixture PRs land, least inflated (238s). Further wall-time gains come from the fixture and clock PRs in this series, not from ordering.

Validation

  • shellcheck scripts/test_packages.sh
  • ./scripts/test_packages.sh lists the same 373 packages as go list ./...; the macOS/Windows pattern yields 45 with cli and enterprise/cli first
  • make test TEST_PACKAGES='./coderd/httpapi/... ./scripts/release' TEST_COUNT=1 passes
  • test-go-pg passed on ubuntu, macOS (BSD awk needed newline-free -v values, fixed), Windows and PG 17 on the measurement commits
  • Pre-commit hook (full make pre-commit, since the Makefile changed) passed

Part of the test-suite speedup series: #29458 (mock clocks), #29459 (notifications golden), #29460 (migrations chain), #29461 (dead code), #29462 (server tailnet DERP poll fix). All independent.

Remote validation

Remote run on a dogfood workspace (separate compute), chat https://dogfood.cdr.dev/agents/5bbe73d0-866a-4841-9cc3-c5e5b4d97e44, tested head ee5d3be95da (and the earlier 37754184d0f): ./scripts/test_packages.sh lists 373 packages, identical set to go list ./..., no duplicates, five heavy packages first; make test TEST_PACKAGES='./coderd/httpapi/... ./scripts/release' TEST_COUNT=1 ran exactly those two packages. Endorsed verdict: PASS.

Xum (an AI agent) authored this PR on behalf of @ibetitsmike.

go test starts packages in argument order, so ./... (alphabetical) left the heaviest packages for the end of the run: in the test-go-pg ubuntu job enterprise/coderd (110s) did not start until t=313s of a 423s run. scripts/test_packages.sh expands the pattern list with the slowest packages first so the -p scheduler packs the long tail behind them.
With every heavy package first, the light packages and their builds formed a 60s tail after the last heavy package finished. Start the five 100s+ packages, let the light packages fill the remaining slots, and run the medium packages once heavy slots free up.
…t count

Three TEST_COUNT=1 runs on this branch: every heavy package first 356s, five heavy then light then medium 364s, ten heavy first 373s, against 364-423s for the last six main runs. The differences are within noise; the five-first order keeps coderd, the future critical path, least inflated by contention.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant