Skip to content

chore(test): De-flake slow and racy tests - #24452

Open
isaacs wants to merge 3 commits into
developfrom
isaacs/de-flake-slow-tests
Open

isaacs wants to merge 3 commits into
developfrom
isaacs/de-flake-slow-tests

Conversation

@isaacs

@isaacs isaacs commented Sep 16, 2026

Copy link
Copy Markdown
Member

Three unrelated flakes, all of which pass locally and fail only under CI load.

Tests that time out loading the SDK

Several unit tests time out in CI on the first test in a file. The first dynamic import of a large SDK graph pays for transforming it, and that cost is charged to whichever test triggers it. Locally it is a few hundred milliseconds. On a loaded runner it crosses the 5000ms test timeout.

Swept every unit test in the repo and fixed the files that show the pattern. Fixed with a bare side-effect import at the top of the file. Collection is not bounded by the test timeout, so pulling the load into it takes the cost off the test without changing what the test exercises.

Exception: nuxt vite/orchestrion has to register its vi.doMock calls before the module under test loads, and a static import is hoisted above them, so it warms the graph from beforeAll instead.

file before after
browser-utils inp 594ms 9ms
browser-utils metricObserverOrdering 409ms 22ms
nextjs appRouterRoutingInstrumentation 750ms 233ms
nuxt vite/orchestrion 536ms 1ms
sveltekit index.workerd 625ms 8ms
tanstackstart-react client/tunnelRoute 433ms 1ms

Which file pays is not fixed: the two browser-utils files traded places between runs, because the transform lands on whichever test reaches the graph first.

Three tests in these packages still run over 400ms, and all three are slow for a reason:

  • The two nextjs appRouter tests busy-wait 1100ms on purpose.
  • conflictingDebugOptions re-evaluates three SDK entry points for each debug-build mock, which no warmup can avoid.

INP span never sent

hidePage waits for one requestIdleCallback before hiding, assuming web-vitals has already queued its own idle callback and will therefore run first. But web-vitals only queues its callback once the click's Event Timing entry reaches its observer, which is after hidePage queues its own. So hidePage wins the race and hides the page too early.

That loses the interaction. The forced report web-vitals does on visibilitychange runs while the metric is still -1, and the pending callback that sets the real value afterwards reports unforced. No INP is emitted at all and the test waits out its 30s timeout.

Fixed by waiting for the entry and queuing the idle callback from a task after it, which puts web-vitals' callback first. Callers that never interacted skip the wait, so the 13 that only need the page hidden pay nothing.

The react-router-6 e2e suite now passes 10/10 and drops from 1.4m to ~53s.

Cron job that never runs

Two bugs in suites/cron/cron, both reproducible under CPU load.

cron's start() sizes the delay to the next tick from two clock reads, clamps a negative result to -1, and then treats that -1 as "stop". A pause between the two reads that straddles a second boundary is enough. The job is left scheduled for nothing, and the scenario exits on its own 15s timer having sent nothing.

That is the reported failure, which reads as 0/5 expected envelope(s) on a clean exit 0 with no captured output. A probe measured that clamp firing 5 times in 613,775 calls under load and never on an idle machine. The scenario now starts the job again, which re-reads the clock away from the boundary, and throws if it still refuses, so a recurrence fails loudly instead of silently.

Separately, each check-in is its own HTTP request, so the order they reach the mock server is not the order the SDK sent them in. Under load the ok check-in overtook in_progress. This is fallout from #20661, which moved these envelopes off stdout, where they were inherently ordered. The assertions are now .unordered().

Failed roughly 1 run in 30 under load before, 120/120 after.

@isaacs
isaacs requested review from a team as code owners September 16, 2026 22:41
@isaacs
isaacs requested review from logaretm, mydea and s1gr1d and removed request for a team September 16, 2026 22:41
@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

size-limit report 📦

Path Size % Change Change
@sentry/browser 29.09 kB - -
@sentry/browser - with treeshaking flags 27.35 kB - -
@sentry/browser - with treeshaking flags tracing without tracing 27.26 kB - -
@sentry/browser (incl. Tracing) 50.6 kB - -
@sentry/browser (incl. Tracing + Span Streaming) 50.62 kB - -
@sentry/browser (incl. Tracing, Profiling) 53.61 kB - -
@sentry/browser (incl. Tracing, Replay) 90.15 kB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 79.25 kB - -
@sentry/browser (incl. Tracing, Replay with Canvas) 94.85 kB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 107.83 kB - -
@sentry/browser (incl. Feedback) 46.62 kB - -
@sentry/browser (incl. sendFeedback) 34.15 kB - -
@sentry/browser (incl. FeedbackAsync) 39.26 kB - -
@sentry/browser (incl. Metrics) 30.1 kB - -
@sentry/browser (incl. Logs) 30.35 kB - -
@sentry/browser (incl. Metrics & Logs) 31.02 kB - -
@sentry/react 30.84 kB - -
@sentry/react (incl. Tracing) 52.94 kB - -
@sentry/vue 36.34 kB - -
@sentry/vue (incl. Tracing) 52.91 kB - -
@sentry/svelte 29.11 kB - -
CDN Bundle 30.8 kB - -
CDN Bundle (incl. Tracing) 51.15 kB - -
CDN Bundle (incl. Logs, Metrics) 33.06 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) 53.14 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) 73.75 kB - -
CDN Bundle (incl. Tracing, Replay) 88.69 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) 90.63 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 94.73 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) 96.78 kB - -
CDN Bundle - uncompressed 91.16 kB - -
CDN Bundle (incl. Tracing) - uncompressed 152.66 kB - -
CDN Bundle (incl. Logs, Metrics) - uncompressed 97.73 kB - -
CDN Bundle (incl. Tracing, Logs, Metrics) - uncompressed 158.61 kB - -
CDN Bundle (incl. Replay, Logs, Metrics) - uncompressed 227.14 kB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 272.23 kB - -
CDN Bundle (incl. Tracing, Replay, Logs, Metrics) - uncompressed 278.17 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 285.93 kB - -
CDN Bundle (incl. Tracing, Replay, Feedback, Logs, Metrics) - uncompressed 291.86 kB - -
@sentry/nextjs (client) 55.27 kB - -
@sentry/sveltekit (client) 51.05 kB - -
@sentry/core/server 39.63 kB - -
@sentry/core/browser 13.66 kB - -
@sentry/node 132.37 kB +0.02% +19 B 🔺
@sentry/node/import (ESM hook with diagnostics-channel injection) 82.03 kB - -
@sentry/node - without tracing 89.82 kB +0.04% +30 B 🔺
@sentry/node - without channel injection 111.22 kB +0.02% +18 B 🔺
@sentry/aws-serverless 98.06 kB +0.03% +24 B 🔺
@sentry/cloudflare (withSentry) - minified 204.69 kB - -
@sentry/cloudflare (withSentry) 509.25 kB - -

View base workflow run

Comment thread dev-packages/test-utils/src/page.ts
@isaacs
isaacs requested a review from a team as a code owner September 17, 2026 00:07
@isaacs
isaacs requested review from JPeer264 and removed request for a team September 17, 2026 00:07

@s1gr1d s1gr1d left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the tests faster!!!

Comment thread packages/browser-utils/test/web-vitals/inp.test.ts Outdated
Comment thread packages/browser-utils/test/web-vitals/inp.test.ts
@isaacs
isaacs force-pushed the isaacs/de-flake-slow-tests branch from d786a06 to e116423 Compare September 17, 2026 15:14
@isaacs isaacs changed the title chore(test): de-flake slow loading tests chore(test): De-flake slow and racy tests Sep 17, 2026
Three unrelated flakes, all of which pass locally and fail only
under CI load.

**Tests that time out loading the SDK**

Several unit tests time out in CI on the first test in a file.
The first dynamic import of a large SDK graph pays for
transforming it, and that cost is charged to whichever test
triggers it. Locally it is a few hundred milliseconds. On a
loaded runner it crosses the 5000ms test timeout.

Swept every unit test in the repo and fixed the files that show
the pattern. Fixed with a bare side-effect `import` at the top of
the file. Collection is not bounded by the test timeout, so
pulling the load into it takes the cost off the test without
changing what the test exercises.

Exception: `nuxt vite/orchestrion` has to register its
`vi.doMock` calls before the module under test loads, and a
static import is hoisted above them, so it warms the graph from
`beforeAll` instead.

| file                                   | before | after |
| -------------------------------------- | -----: | ----: |
| browser-utils inp                      |  594ms |   9ms |
| browser-utils metricObserverOrdering   |  409ms |  22ms |
| nextjs appRouterRoutingInstrumentation |  750ms | 233ms |
| nuxt vite/orchestrion                  |  536ms |   1ms |
| sveltekit index.workerd                |  625ms |   8ms |
| tanstackstart-react client/tunnelRoute |  433ms |   1ms |

Which file pays is not fixed: the two browser-utils files traded
places between runs, because the transform lands on whichever
test reaches the graph first.

Three tests in these packages still run over 400ms, and all three
are slow for a reason:

- The two nextjs `appRouter` tests busy-wait 1100ms on purpose.
- `conflictingDebugOptions` re-evaluates three SDK entry points
  for each `debug-build` mock, which no warmup can avoid.

**INP span never sent**

`hidePage` waits for one `requestIdleCallback` before hiding,
assuming web-vitals has already queued its own idle callback and
will therefore run first. But web-vitals only queues its callback
once the click's Event Timing entry reaches its observer, which
is after `hidePage` queues its own. So `hidePage` wins the race
and hides the page too early.

That loses the interaction. The forced report web-vitals does on
`visibilitychange` runs while the metric is still -1, and the
pending callback that sets the real value afterwards reports
unforced. No INP is emitted at all and the test waits out its 30s
timeout.

Fixed by waiting for the entry and queuing the idle callback from
a task after it, which puts web-vitals' callback first. Callers
that never interacted skip the wait, so the 13 that only need the
page hidden pay nothing.

The react-router-6 e2e suite now passes 10/10 and drops from 1.4m
to ~53s.

**Cron job that never runs**

Two bugs in `suites/cron/cron`, both reproducible under CPU load.

`cron`'s `start()` sizes the delay to the next tick from two
clock reads, clamps a negative result to -1, and then treats that
-1 as "stop". A pause between the two reads that straddles a
second boundary is enough. The job is left scheduled for nothing,
and the scenario exits on its own 15s timer having sent nothing.

That is the reported failure, which reads as `0/5 expected
envelope(s)` on a clean exit 0 with no captured output. A probe
measured that clamp firing 5 times in 613,775 calls under load
and never on an idle machine. The scenario now starts the job
again, which re-reads the clock away from the boundary, and
throws if it still refuses, so a recurrence fails loudly instead
of silently.

Separately, each check-in is its own HTTP request, so the order
they reach the mock server is not the order the SDK sent them in.
Under load the `ok` check-in overtook `in_progress`. This is
fallout from #20661, which moved these envelopes off stdout,
where they were inherently ordered. The assertions are now
`.unordered()`.

Failed roughly 1 run in 30 under load before, 120/120 after.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@isaacs
isaacs force-pushed the isaacs/de-flake-slow-tests branch from ae8b0f4 to 015bdd0 Compare September 17, 2026 15:20

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 63f0621. Configure here.

@JPeer264 JPeer264 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MVP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants