Skip to content

test(coderd/database/dbtestutil): disable Postgres JIT for test databases - #29465

Open
ibetitsmike wants to merge 1 commit into
mainfrom
mike/test-dbtestutil-jit-off
Open

ibetitsmike wants to merge 1 commit into
mainfrom
mike/test-dbtestutil-jit-off

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

On near-empty test databases the planner misestimates some queries so badly that they exceed jit_above_cost, and Postgres then LLVM-compiles them on every call. For GetUserStatusCounts that is about 275ms of a 290ms execution (EXPLAIN (ANALYZE) on an empty database estimates 5.7M rows for the dates_of_interest x relevant_status_changes nested loop; actual rows: 61). Multiplied by the 925 leaves of TestGetUserStatusCounts, that single test was 85% of the coderd/database package time.

JIT is an execution optimization with no semantic effect, so dbtestutil.NewDB now runs ALTER DATABASE <db> SET jit = off on each per-test database, next to the existing timezone setting. Database-level settings are not copied by CREATE DATABASE ... TEMPLATE, so this has to run per clone, which is where the timezone statement already runs; the CODER_PG_CONNECTION_URL path goes through the same function.

Before / After

Local, go test -count=1 -tags=testsmallbatch -parallel=8 (shared host, load about 50):

Before After
./coderd/database/ -run '^TestGetUserStatusCounts$' ok 63.2s ok 6.4s
./coderd/database/ ok 57.2s ok 10.0s
./coderd/database/dbauthz/ ok 7.4s / 14.0s ok 10.3s / 7.6s (noise)
./coderd/database/dbpurge/ ok 4.1s ok 3.8s
./coderd/httpmw/ ok 1.3s ok 1.0s

Only coderd/database (172s in the baseline CI job) is expected to move materially; the other packages are listed to show no regression.

Validation

  • The five runs above pass; TestGetUserStatusCounts still runs its 925 subtests
  • golangci-lint run ./coderd/database/dbtestutil/... with a fresh cache: 0 issues
  • Repository pre-commit hook passed on the commit

Production note, not addressed here: with jit=on (the default in the pgdg builds) production pays the same per-call compilation for GetUserStatusCounts; the query shape (generate_series cross join with no statistics on the CTE) deserves its own fix.

Part of the test-suite speedup series (see #29457). Independent of the other PRs; #29464 (fixture sharing in the same test) composes with it.

Remote validation

Remote run on a dogfood workspace (separate compute), chat https://dogfood.cdr.dev/agents/5bbe73d0-866a-4841-9cc3-c5e5b4d97e44, tested head 353879f5d12 against base 9d973372774 on the same machine (its base time, 56.4s for the package, matches the table above): ./coderd/database/ -run '^TestGetUserStatusCounts$' 46.97s to 13.81s (925 subtests pass on both); ./coderd/database/ 56.43s to 20.49s (32.48s on a first run); dbauthz, dbpurge, httpmw unchanged; -race -count=2 -shuffle=on: pass. A live pg_db_role_setting query during a test run showed {TimeZone=...,jit=off} on every per-test database and nothing on the tpl_ template, as intended. The standalone speedup there is 3.4x for the test and about 2x for the package, less than the 10x and 5.7x in the table above, which came from a single run on the development host: on the dogfood host the per-test database clone dominates once JIT is off. The headline numbers (test 0.6s, package 7.5s) are reached together with #29464, which removes the per-leaf clone. Endorsed verdict: PASS on behavior; the table above overstates the standalone gain.

Xum (an AI agent) authored this PR on behalf of @ibetitsmike.

…ases

On near-empty test databases the planner misestimates some queries so
badly that they exceed jit_above_cost, and Postgres LLVM-compiles them on
every call. GetUserStatusCounts alone spent about 275ms of its 290ms
execution in JIT. JIT is an execution optimization with no semantic effect,
so disable it per test database next to the existing timezone setting.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-17T07:48:41.103423Z 353879f Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant