[pull] main from triggerdotdev:main - #17

…rk (#4756)

Fixes found while reviewing #4547, stacked on that branch so they can be
reviewed on their own and merged into it. One commit per fix.

## Write-path correctness

**Refuse account writes while impersonating.** The five
`dashboardPreferences` writers already no-op for an impersonating admin,
but the three profile writers added next to them did not, and
`requireUserId` returns the impersonated user's id. Both gates now
refuse up front and say so, rather than the preference writers silently
no-opping while the page reports success.

**Preserve unknown keys on a full-blob write.**
`mutateDashboardPreferences` parses the JSON column, hands the result to
a mutator and persists the whole object back. zod strips keys it does
not declare, so a deploy that predates a preference field drops it on
the next write through that path — and
`updateCurrentProjectEnvironmentId` sits on the navigation hot path.
`preserveUnknownKeys` re-attaches them at the write. Note this cannot
help deploys already running, so it makes this the last release able to
strip rather than retroactively protecting the fields added in #4547.

**Scope hidden-sidebar writes to what was shown.** The customize dialog
builds its hidden map from the sections it can see and the write
replaced `hiddenItems` wholesale. The profile page has no org in scope,
so it resolves sections from the most-recently-updated project's org:
confirming there dropped hidden ids belonging to sections that org's
flags exclude. The payload now carries the ids the dialog rendered and
the write only replaces those. Submissions without the list stay
authoritative.

**Consider both addresses when checking email ownership.** The check
only looked at the address the user already had; it now considers the
current and submitted address together, so an org managing either one
governs the change. Validation moved ahead of the check, and
`emailDomainOf` splits on the last `@`.

## Interaction

**Revert unsaved themes, debounce contrast saves.** The theme and
system-theme selects stamp `data-theme` before the write lands. When it
fails, the loader returns the value it always had — so
`useSystemThemeSync`'s effect deps are unchanged and React's vdom diff
sees no change either, and nothing rewrites the attribute. The page kept
rendering a theme that was never stored while the select showed the
stored one. The stored pair is now re-applied explicitly, as the side
menu's switcher already did. The contrast slider is debounced because
Radix commits on every arrow keypress, so a keyboard user crossing the
range fired one write per step.

**Tick More options for themes outside the short list.** The appearance
submenu offers System, Light and Dark; Black and White live on the
profile page. With one of those stored, every row read as unselected.

## Subtraction

**Drop the profile update rate limiter.** It covered one of four paths
that write the same column — `resources.preferences.sidemenu` and
`.favorites` take unlimited authenticated writes and go through the
locked read-modify-write, which is more expensive than the single narrow
`jsonb_set` this capped. It was also what made the contrast slider
unusable by keyboard. If preference writes want limiting, it belongs in
one place covering all of them.

**Resolve email ownership when the dialog opens.** It fans out one SSO
status lookup per organization the user belongs to and ran in the
profile loader on every page view, purely to pick which body the dialog
renders. The action re-derives it before writing either way, so the
check that guards the write now has one call site instead of two.

## Testing

`typecheck --filter webapp` and `lint` clean. New unit tests for
`preserveUnknownKeys`, `mergeHiddenItems` and `emailDomainOf`;
`themePreference`, `mergeHiddenItems` and `ssoManagedIdentity` suites
pass locally (26 tests). The rest of the webapp suite needs
testcontainers and is left to CI.

No changeset or `.server-changes` entry: everything here fixes code on
the parent branch that has not shipped. The one exception worth a
maintainer's call is `mergeHiddenItems`, which also touches the side
menu's own customize path.

…4751)

## Summary

On a self-hosted instance, saving anything on the global admin feature
flags page also deleted the two read-only flags,
`defaultWorkerInstanceGroupId` and `taskEventRepository`. Losing the
first one leaves deployed runs with no default worker group. Neither
deletion showed up in the confirm dialog, so the flags disappeared
silently.

## Root cause

The page submits only the flags its UI is managing, and strips the
read-only ones from the payload unless "Unlock read-only flags" is
ticked. The action treated every catalog key absent from that payload as
"the admin unset this", and protected the locked keys only when the
instance was managed cloud. Anywhere else, both locked rows fell
straight into the delete sweep.

The protection now keys off what the client says it was editing rather
than off the deployment:

```ts
const canDeleteLocked = params.unlockLockedFlags && !params.isManagedCloud;
...
} else if (canDeleteLocked || !GLOBAL_LOCKED_FLAGS.includes(key)) {
  keysToDelete.push(key);
}
```

Exactly one case changes: a locked flag, on a non managed-cloud
instance, with the flags not unlocked, is now kept instead of deleted.
Managed cloud behaviour is bit for bit identical, and ticking the unlock
box still gives a self-hosted instance full control. The write moves
into `replaceGlobalFeatureFlags` so it can be driven directly in tests
against a real Postgres.

…Postgres waitpoint implementation (#4753)

Extracts every Postgres waitpoint and edge operation out of
`WaitpointSystem` into a `WaitpointCoordinator` seam with one Postgres
implementation, so a different coordination backend can be plugged in
later without any caller changing.

Pure refactor. Zero behaviour change, and zero test-file diffs — the
existing engine corpus is the characterisation test.

## What moved

`WaitpointCoordinator` (`waitpointCoordinator/types.ts`, declared with
`type`) has nine members: `clearRunBlockState`, `readRunBlockState`,
`registerBlocks`, `registerBlocksLockless`, `complete`,
`createDateTimeWaitpoint`, `createManualWaitpoint`,
`mintAssociatedWaitpointData`, `createAssociatedWaitpoint`.

`LegacyPostgresWaitpointCoordinator` implements them against the run-ops
store. Its dependencies are `{ runStore, prisma, logger }` only, so it
structurally cannot reach the run lock, the worker, or the event bus —
orchestration stays in `WaitpointSystem`, which keeps all ten public
signatures, all six `worker.enqueue` sites, the racepoints, the snapshot
transitions, and the event emissions.

Two register methods rather than one with a flag, so "the batch path
issues no extra query" is structural instead of conditional. Both share
one private edge-write helper.

## Six notes for reviewers — please read before "simplifying" any of
these

1. **`nanoid(24)` is called twice with different values on purpose**, in
each create path: once for the upsert `where` key, once for
`create.data`. Hoisting either to a shared constant makes the where-key
match the create-key, turning a guaranteed-miss upsert into a possible
update. In `createManualWaitpoint` both calls plus
`WaitpointId.generate()` stay *inside* the retry loop so each attempt
tries a fresh key.

2. **The two enqueue conditions are deliberately asymmetric.** DATETIME
enqueues `finishWaitpoint` unconditionally after a non-cached create,
with `availableAt: completedAfter`. MANUAL enqueues only when `timeout`
is set. That is existing behaviour, not an oversight. The coordinator
returns a discriminated union on `kind` rather than a boolean so the
enqueue is structurally unreachable on the cached path.

3. **One false clause was deleted from a moved comment.** The old
comment on the full-clear delete claimed the caller's `tx` is not
forwarded. The code does forward it, and `PostgresRunStore` uses `tx ??
this.prisma`, so a single store joins the caller's transaction — only
the routing store strips it. The rest of that comment is unchanged.

4. **The MANUAL timeout enqueue now sits outside the P2002 retry loop.**
Safe because the worker is Redis-backed and cannot raise
`Prisma.PrismaClientKnownRequestError`, so the loop never retried on it.
**If a Postgres-backed enqueue is ever swapped in, that equivalence
breaks silently.**

5. **The coordinator caches `runStore`/`prisma`/`logger` at
construction**, where the old code read `this.$.*` per call. Equivalent
only because nothing reassigns them: one assignment at
`engine/index.ts`, and the `resources` object is a `const` that is never
mutated.

6. **Two comments in other files are now stale and were left alone** —
`engine/index.ts` and `completeWaitpointCrossSeamGuard.test.ts` both
describe routing as the first statement of
`waitpointSystem.completeWaitpoint`. Both tests still pass, because that
guard sits in `index.ts` before the delegation. Left untouched to keep
this diff to three files.

## Preserved verbatim

The `unnest` edge CTE rather than a `Waitpoint` join; the pending count
as a separate statement after the edge write (READ COMMITTED needs its
own snapshot); completion's `findWaitpointOnPrimary` re-read through the
*resolved handle* while the blocked-run fan-out goes back through the
*router*; the residency and colocate hints, with colocation objects
built only in the Postgres arm and the count keeping its `runId`
argument; `ON CONFLICT DO NOTHING` and the `(taskRunId, waitpointId,
batchIndex)` multi-index edge semantics; the unread `batchId` select,
which rides inside two `logger.debug` payloads.

`internal-packages/run-store/` is untouched, so the CTE and the conflict
semantics never moved.

## Verification

| Check | Result |
| --- | --- |
| Engine corpus | 61/61 files, 353 passed, 1 skipped, **0 failed**
(baseline: 352 passed, 1 failed) |
| Test-file diffs | **empty** |
| `run-engine` typecheck | `tsc --noEmit -p tsconfig.build.json` exits 0
|
| `webapp` typecheck | 146 errors on this branch, **146 identical errors
at baseline** — pre-existing, none added |

The webapp typecheck does not pass. The failures are pre-existing
(`PrismaPg` not assignable to `never`; missing `@trigger.dev/rbac`
exports) and the sorted error lists are byte-identical to the merge
base, so this branch adds none — but the criterion is genuinely unmet
and needs a separate fix.

No changeset and no `.server-changes` note: internal refactor with no
user-visible change.

## Follow-ups this surfaced

- The dominant RUN waitpoint is still created outside the seam —
`buildRunAssociatedWaitpoint` now mints through the coordinator, but the
row is inserted nested inside `createRun`/`createFailedRun`. That needs
its own packet before a second backend lands, or the commonest waitpoint
gets split across two of them.
- `clearRunBlockState` overloads opposite outcomes on `undefined` versus
`[]`: `undefined` clears every edge, `[]` clears none. Both callers are
correct today; worth splitting when the file is next touched.
- A stray non-`.sql` entry in `internal-packages/clickhouse/schema/`
breaks every `containerTest` in the repo, because the testcontainers
migration reader `readFile`s every `readdir` entry without filtering
despite a comment claiming it filters. Hit this during setup; unrelated
to this change and left for a separate fix.

…le settings via users.d (#4762)

Carries over the self-hosted ClickHouse fix from #4546 by @Leafgard,
whose commits are preserved here, plus follow-up polish. Opened in-repo
because the fork is org-owned, which GitHub's "Allow edits from
maintainers" doesn't cover.

fixes #4343

## What was wrong

Two independent problems in `hosting/docker/clickhouse/`:

1. **The `<profiles>` block never applied.** It sits in `override.xml`,
mounted under `config.d` - but ClickHouse only reads profile settings
from the users config tree. Verified on the pinned image: before this
change `max_block_size` sat at its default `65409` with `changed=0`, so
the advertised low-memory settings had never taken effect at all.
2. **Every ClickHouse system log table was enabled and unbounded.** On a
sub-16GB machine their background merges outgrow the memory cap;
ClickHouse's [low-RAM
guide](https://clickhouse.com/docs/operations/tips) recommends disabling
them. The dev stack already does this - `hosting/docker` never got it.

## What this does

- `clickhouse/override.xml`: disables the high-frequency telemetry
tables, and bounds the ones worth keeping with a config-level `<ttl>` -
`query_log` and `part_log` at 7 days, `error_log` at 30. A config-level
TTL survives log-table recreation, unlike `ALTER ... MODIFY TTL`.
- New `clickhouse/users-override.xml`, mounted at
`users.d/override.xml`: carries the profile settings so they actually
apply, completes the sub-16GB set with `max_threads=1`, and zeroes the
memory/query profilers, whose samples were the main source feeding
`trace_log`.
- `webapp/docker-compose.yml`: adds the `users.d` mount.

## Verification

Ran `clickhouse/clickhouse-server:26.2` with these exact mounts, and
`25.12` to cover the documented 25.8 floor:

- All 9 profile settings report `changed=1`, and a custom
`CLICKHOUSE_USER` inherits them.
- `users.d` merges rather than replaces: the `default` user, its
password, `access_management` and the `readonly` profile all survive, so
the compose healthcheck still passes.
- `remove="1"` is a clean no-op on keys absent from a given version - no
empty section, no accidental table, no startup error - so pinning
`CLICKHOUSE_IMAGE_TAG` to an older supported tag won't crash-loop.
- TTLs land in the real DDL: `TTL event_date + toIntervalDay(7)` /
`(30)`.
- In-place upgrade on a populated volume: clean restart, data preserved,
and ClickHouse lazily renames the pre-existing `query_log`/`error_log`
to `query_log_0`/`error_log_0` as it applies the new retention.

## Notes for review

- **`part_log` is kept (bounded) rather than disabled.** It appears in
neither report behind this change and isn't on ClickHouse's sub-16GB
list, but it's the merge history you'd need to diagnose a recurrence.
Measured at ~0.18 KiB per part event under insert churn - about 10x
cheaper than `text_log` over the same window - so a TTL bounds it rather
than removing it.
- **The profile settings go live for the first time here.** On larger
machines that's a real, intended throughput change: `max_threads=1`,
`max_download_threads=1`, parallel parsing and formatting off.
- **Disabling a log table stops new writes but doesn't delete existing
data.** Reclaiming disk on an existing deployment needs `DROP TABLE
system.<name> SYNC`, including the `*_log_0` leftovers.

## Known gaps, deliberately not in this PR

- The Helm chart carries the same ineffective `<profiles>` block in
`values.yaml` and mounts nothing into `users.d`, so this fix isn't
currently expressible there.
- `background_schedule_pool_log` is enabled by default with no TTL and
is disabled by neither stack.
- The dev stack's disable list has drifted from this one.
- The compose healthcheck still logs a query every 5 seconds.

---------

Co-authored-by: Yann SEGET <yann.seget@actemium.ch>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…d resolver contract (#4760)

Builds on
[#4754](#4754), which
added the store this contract belongs to.

## Why

Two migrations are moving to Redis in parallel, and execution snapshots
reference completed waitpoints across the boundary between them. If the
record shape is agreed only once both halves are built, the correction
lands mid-rollout: dual-write is live, real keys are in Redis, and
changing the entry format then means two versions of the entry
coexisting plus a migration for whatever was already written. Agreeing
it now, while nothing writes a pointer, makes that same correction a
type edit.

The reserved-and-empty field is the same argument one level down. The
entry format is what dual-write writes, so adding a field to it later
splits the format in two. Reserving it before any write means the format
never changes after writes begin.

## Summary

Adds the type contract for carrying completed waitpoints alongside the
Redis-backed execution-snapshot store: a `{cycleSeq, count}` pointer on
the snapshot entry, the record shape that pointer resolves to, and the
read-time resolver signature. Nothing constructs or reads a pointer yet,
so this is inert on merge.

The record shape has to reproduce
`enhanceExecutionSnapshotWithWaitpoints` field for field, because that
is what the executor consumes. A conformance test runs the real function
against a reference resolver over an exhaustive grid of 6144 input
combinations, derived from every `Waitpoint` column the function reads
rather than hand-picked.

## Design

`completedWaitpoints` is reserved on the entry type and always unset.
`append()` rejects a set value, because the pointer's physical home is
the `<snapshotId>#c` sidecar field rather than the entry JSON. The
append script mints both halves after the client serializes the entry,
and the entry JSON has to stay byte-identical to the Postgres row so the
two can be compared during a dual-write rollout.

Two rules are worth calling out, both found by making the test fail
rather than by reading the code:

* `records` is the authoritative waitpoint set, not `order`. Only batch
waits carry an index, so `order` is empty for a single `triggerAndWait`
while the Postgres join still holds the id. Comparing id sets over
`order` would serve the previous wait cycle's records.
* `deriveFromRun` requires a non-null `completedByTaskRunId`.
`Waitpoint.completedByTaskRun` is `onDelete: SetNull`, so an orphaned
RUN waitpoint keeps its output with no run left to derive from. Those
records carry their output inline instead.

`tsconfig.freeze-test.json` typechecks the conformance test, which the
package build config excludes. Without it, renaming a field in the
frozen type compiles clean and every test stays green, so the literal
assertions in the test would only pin the test's own writer.

## Fixes carried along

Auditing the contract surfaced three defects in the append script, each
with a regression test that fails when the fix is reverted:

* A new wait cycle now clears any `records` left on a reused key. A
`seq` counter lost to eviction can re-mint a `cycleSeq` whose key still
holds another cycle's records, and `order` and `count` are overwritten
together, so the mismatch check could not see the drift.
* A carry-forward now attaches a pointer only if the current keyspace
incarnation actually minted that cycle. The previous key-exists check
adopted a dead incarnation's records under a count that agreed with
them, reporting no mismatch.
* The cycle-key size metric now counts `records`, not only `order`. It
reported 7 bytes for a 20 KB key, so the high-water log could never fire
on the field that grows.

…#4755)

## Summary

Adds the shard-selection stage of run-id minting.
`resolveMintShard(env)` returns which run-ops database an environment
mints its new run roots into: the active shard list, then a fleet-wide
override, then a per-environment or per-organization pin, then a
rendezvous hash of the environment id.

That half is inert. Nothing calls `resolveMintShard`, no deployment has
any of the new flags set, and an empty active list returns the current
answer without reading anything.

**The other half is not inert, and it is where review effort belongs.**
To stamp a grace window this needs a read-then-write under a lock, so it
rewrites the global feature-flag write path that `runOpsMintKind`
already depends on in production. See below.

## Placement

Resolution reads the active list from a global flag, applies the grace
window, and then picks:

- a fleet-wide override if one is set, which is how a cutover completes
without visiting each organization. `new` holds the whole fleet on the
current id format.
- otherwise a per-environment or per-organization pin. `new` holds one
organization back while the rest move, which is how a canary works.
- otherwise a rendezvous hash, so adding a shard moves only about
1/(N+1) of environments and removing one moves only its own.

Two hash details are load-bearing. Scores are 64-bit `sha256(envId \0
key)`, because a 32-bit score collides at our environment count and an
undetected tie would resolve by iteration order. The parsed key list is
sorted, because otherwise two deployments listing the same shards in a
different CSV order would place environments differently.

A pin or override naming a shard that has left the active list falls
through to the hash and reports once. Honouring it would leak the drain
the active list exists to perform, and throwing would fail triggers
whenever a pinned shard drains.

## Why the active list is a flag and not an environment variable

A deploy rolls for hours, so two pods hold two different environment
values at the same time. A list held in the environment therefore splits
the fleet for the length of the rollout, with new pods placing an
environment on one shard and old pods on another. A grace window
measured in seconds cannot cover that, and the same knob times the
existing mint-kind flip so it cannot simply be lengthened. An
environment variable also cannot record its own flip time, and an
operator cannot know a rollout's end in advance.

So the list, its grace stamp and the override are global flags, written
server-side against the control-plane clock under an advisory lock. This
branch adds no environment variables.

## The write path, which is live

Stamping generalises to any number of graced flag groups in one
transaction under one lock. That has three consequences a reviewer
should look at directly:

- It closes a real bug. `runOpsMintKind` is an editable control on the
global flags page, and that page previously wrote it with a bare upsert:
no lock, no stamp. An operator flipping mint kind through the UI got an
ungraced flip, so every pod crossed the cutover at a different moment.
Verified against a running instance, before and after.
- A graced group is all-or-nothing. Submitting its primary writes the
group with a fresh stamp; omitting it deletes the primary and its stamp
together, because a stamp left without its primary keeps being served
and would mint into a shard just removed.
- The advisory lock takes the previous id as well as the current one, in
a fixed order, so writers on an older release still serialise during a
rollout. The legacy id can be dropped one release after this ships.

This folds with #4751 rather than replacing it: its `unlockLockedFlags`
rule decides what the sweep may delete, and the graced groups keep their
stamp under the lock. Both sets of tests pass.

## Notes for review

Determinism is a property of the pure core for fixed inputs. The wrapper
supplies the clock, the same split `effectiveMintKind` already uses. A
failed read of the list falls back to the current id format rather than
guessing.

Six flags appear in the admin pages immediately. The two pins are
per-organization, so they render read-only on the global page. The list,
its stamp and the override are deployment-wide, so they render read-only
in the organization dialog.

Nothing bounds the active list against shards that actually exist. That
is safe while nothing mints, but the change that carries a shard key
into an id must land after the shard descriptors bound the list, or
bound it itself.

…d waitpoint ids (#4761)

Builds the Redis-backed half of the waitpoint coordinator, beside the
Postgres coordinator that #4753 extracted. Adds the coordination
protocol as Lua scripts, the run-ops-format waitpoint id scheme, and the
key layout. **No caller wires any of it up.**

Refs TRI-13440.

## Inert by construction

Merging this changes nothing observable. 3180 insertions, **zero
deletions**, nine new or additively-edited files.

- `WaitpointStoreCoordinator` is never constructed outside its own tests
and the benchmark.
- No env var, no config plumbing, no connection. It takes `redisOptions`
as a constructor argument.
- `waitpointSystem.ts` is untouched. Every live waitpoint operation
still runs on Postgres through the coordinator merged in #4753.
- No changeset and no `.server-changes` note — nothing here is
user-facing yet.

Deploying this needs no Redis or MemoryDB instance. That becomes a
prerequisite when a later change routes traffic onto the store behind a
per-organisation flag.

## What's here

**Nine Lua scripts**, each atomic on one hash tag. Seven mutate state —
create-if-absent, register-or-report, complete, idempotency reserve,
absorb, deliver, clear. One reads state (`runReadBlockState`) and is
separate because the pending, delivered and edge sets must be read as
one consistent view. One discards an idempotency loser.

**Two hash tags, deliberately.** `wp:{waitpointId}` holds a waitpoint's
record, status, completion envelope and watcher hash. `wp:run:{runId}:*`
holds one run's pending set, delivered set and edge set. A waitpoint has
N watchers, so it cannot live under any single run's tag.

**Waitpoint ids** reuse the run-ops body layout: a 24-char base32hex
core, a type char (`r`/`b`/`d`/`m`), and version char `w`. RUN and BATCH
ids derive from their anchor's core, so create-if-absent is idempotent
with no lock. `parseWaitpointId` is total and never throws.

**The single-slot guard.** Every script invocation goes through one
private wrapper that asserts all keys share a hash tag. A single-node
test server accepts what a real cluster rejects, so this assertion is
the only enforcement — and it is mutation-tested: removing it fails a
test.

## Measured

Against the same population of real Postgres rows:

| | store | postgres |
|---|---|---|
| pending count (the blocked/unblocked gate) | 0.13 ms p50 | 3.32 ms p50
|
| full-payload read | 1.45 ms p50 | 7.70 ms p50 |

Both are lower bounds: the benchmark charges Postgres a `COUNT(*)`,
while the resume-time read is a join with a partial select plus
filtering in JavaScript.

Store-only paths, no Postgres counterpart: block+complete+deliver 0.88
ms p50; 100-watcher fan-out 13.8 ms; a 1001-edge fan-in 149.8 ms, flat
at 0.15 ms per edge and round-trip bound rather than algorithmic.

The benchmark lives in `*.bench.test.ts` and is excluded from the
default suite.

## Review notes

- **The type surfaces are not reconciled yet, on purpose.** `types.ts`
(from #4753) carries the coordinator interface; `storeCoordinator.ts`
declares its own operation types because this was built in parallel. The
wiring change reconciles them.
- **The read-time resolver is not here.** Another lane froze its
contract while this was in flight, and its frozen types are not yet on
main. Building a second copy would fork a just-frozen contract.
- **Teardown is one-shard while registration is two-shard.** A terminal
clear leaves a run registered as a watcher on the waitpoints it was
blocked on, because the watcher hash is under a different tag and no
script may span slots. Recorded, not fixed here — it needs a retention
decision, and nothing observes it while the code is unwired.

## Verification

79 tests in the coordinator suite, 58 in the id suite. `typecheck` on
run-engine and webapp, `build` on core, `knip`, `oxfmt` and `oxlint` all
clean. The engine corpus passes 82/82.

Every invariant is mutation-tested rather than merely asserted. A
whole-branch review ran 14 mutants and killed 12; the two survivors were
fixed with their own mutation checks.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Adds an experimental `--local-bundle` flag to native build deployments:
the project is installed and bundled on the local machine (exactly like
in the depot path) and only the resulting build context is uploaded. The
remote build then runs just the container image build.

### Design

- The uploaded artifact is the same build context classic deploys
produce: bundled output, a synthesized package.json with the resolved
externals, build.json, and the generated Containerfile. The bundle is
secret-free: build.json is deliberately scrubbed because it is copied
into the image, and build-arg values never enter the bundle at all.
- Build-arg values are sent with the deployment initialization request
instead, stored encrypted (aes-256-gcm) in a new
`WorkerDeployment.buildEnvVars` column, and cleared on every terminal
status transition. They exist at rest only for the active build window,
always encrypted.
- A dedicated `GET /api/v1/deployments/:id/build-env-vars` endpoint
returns the decrypted values to the same principals that can already
read the environment's variables. It answers with an empty record for
deployments without stored values or in a terminal state, keeping secret
access to a single auditable route.
- Size limits are enforced server side and pre-checked client side. If
the server does not acknowledge storing the values, the CLI fails fast
instead of letting the remote build run without them.
- A `--from-bundle <dir>` mode builds a deployment image straight from
such a bundle directory, skipping config loading and bundling entirely.
In attach mode it fetches the stored build-arg values through the new
endpoint.
- Env var syncing (the `syncEnvVars` extension) happens client side,
before the deployment initializes, since the remote side never sees the
unscrubbed manifest.
- Bundle artifacts use a distinct type and storage prefix so the server
can always distinguish them from source uploads.

Default setup doesn't run CodeQL on pull requests from forks, so
external contributions are stuck on PR checks that never come. Advanced
setup fixes this.

Languages, categories and `main` coverage match the current default
setup. The bare `pull_request` trigger (no `branches` filter) keeps
stacked PRs scanned, whose base isn't `main`.

Default setup has to be disabled in Settings -> Code security for these
uploads to be accepted. Until it is, the CodeQL check here fails with
`CodeQL analyses from advanced configurations cannot be processed when
the default setup is enabled`.

## Summary

Improves the performance and reliability of the runs list and the
`runs.list` API, especially for large projects and filtered views.

## What changed

- **Filtered runs-list queries use `PREWHERE`.** Immutable and
additive-only filters (tags, task identifier, version, queue, region,
machine, and the rest) are applied in `PREWHERE` on the `task_runs_v2
FINAL` scan, so ClickHouse filters, and uses the tags skip index, before
it reconciles versions and materialises the wide columns. Same results,
far less memory per query. `status` stays in `WHERE`: it changes across
a run's versions, so filtering it before `FINAL` could return stale
rows.
- **The runs-list ClickHouse pool gets per-query guardrails**, all
env-configurable: a `max_execution_time` paired with the client request
timeout, a per-query `max_memory_usage`, a `max_threads` cap, and
`readonly`. Each bounds a single query to itself, so a heavy query can't
affect other queries, and they are safe as pool-level settings only
because this pool is read-only.
- **Billing and bulk count reads move to the read pool**, off the write
pool.

Defaults are conservative for self-hosters; production values are set
via env.

…ids (#4770)

Skew protection resolves a run's worker by (environmentId, externalId,
status=DEPLOYED). A miss parks the run and then expires it, so
deployments
predating the feature — which already carry the same value in commitSHA
— need
externalId populated to stay reachable. Vercel instant-rollback is the
sharpest
case, which is why the scope is the current promotion plus a recent
window
rather than current alone.

Follows the existing backfill shape: admin PAT, keyset cursor over
environments,
per-environment action results, pMap, dryRun defaulting to true. Reuses
normalizeExternalDeploymentId so a backfilled id is byte-identical to
what a
build writes, and the update re-checks externalId IS NULL so a deploy
landing
mid-backfill keeps its own id.

Refs TRI-13464.

…d shared test utilities (#4772)

## Summary

Adds the read comparator for the in-progress migration of the run
execution-snapshot log from Postgres to Redis. The comparator samples a
single read against both stores, normalizes the two results to one
shape, and reports any per-field difference with a tagged metric. It
never serves a read itself: the diff layer imports only types, so it
cannot hold a store client, and a test enforces that by failing if any
value import appears.

Also adds a combined Postgres-and-Redis test fixture and two shared test
utilities (a cluster-slot assertion and a generic fault-injection
harness) that the parallel Redis-store work reuses.

Everything here is inert. Nothing constructs the comparator, so merging
changes no runtime behavior. It becomes active only when a later change
turns on compare mode.

## Notes

The divergence classes separate real differences (scalar, ordering,
waitpoint id set, validity, missing on one side) from two expected
classes that must not be driven to zero: a rotated idempotency key, and
a Redis-only surplus at a since-cursor tie. The since comparison is
direction sensitive: a Postgres-only entry at the cursor is always a
lost write, never an expected tie.

Adds an `ADMIN_DASHBOARD_ENABLED` env var (default: enabled) that turns
the admin dashboard and user impersonation off for an entire instance.

When disabled:
- every admin dashboard page redirects away, and the admin navigation
isn't rendered
- existing impersonation cookies are ignored, and any lingering session
is actively terminated with an audit record
- every flow that could start an impersonation responds 404, and no
impersonation tokens are minted

Stopping an impersonation always works regardless of the flag, so
nothing gets stuck. Machine-to-machine admin API endpoints are not
affected. The variable is documented for self-hosters; instances that
don't set it are unaffected.

…when a runs list query is too expensive (#4773)

## Summary

When a runs list query is too expensive to complete, it now fails with a
clear, actionable error instead of a generic 500.

Previously, a runs list query that exceeded ClickHouse resource limits
threw an opaque error. On the public `runs.list` API that surfaced as a
retryable 500, so a customer task calling it would keep retrying a query
that could never succeed. On the dashboard it rendered as a generic
error page with no hint about what to do.

## Fix

The ClickHouse client now tags resource-limit failures (memory, time,
rows, bytes) with their error type, and the runs repository maps those
to a dedicated `RunsListQueryError` (HTTP 422).

- `runs.list` API returns 422 with a message telling the user to narrow
their `created_at` range, plus an `x-should-retry: false` header so the
SDK does not retry it.
- The dashboard runs list (and the errors, scheduled, standard-task,
agents, and webhooks list views) render a shared error state with the
same guidance, so a too-broad time filter is recoverable by the user.

Switching between deployments in the dashboard re-fetched the whole
build log stream from record zero and re-rendered the list line by line
every time. Logs are now cached per deployment for the lifetime of the
tab: revisiting a deployment shows its logs immediately, and the stream
is resumed from the next unread record rather than restarted. Finished
deployments whose stream has been read through the `finalized` event are
served entirely from the cache.

### Changes

The stream/cache logic moved out of the route into a `useDeploymentLogs`
hook. On each deployment switch it seeds state from the cache, resumes
the S2 read session at `nextSeqNum`, and writes back on cleanup or
natural session end. Completion is derived from the stream's own
`finalized` event (plus a terminal deployment status), not from the
session closing, so a session cut short by token expiry or a proxy
cannot pin a truncated log in the cache.

Memory is bounded by a small LRU (`deploymentLogsCache`): at most 20
deployments and 20,000 log lines in total, least recently viewed evicted
first. The most recently viewed deployment is always kept, so a single
very large log can temporarily exceed the line budget on its own.
Records are batched into one state update per tick instead of one per
line.

## What

Makes `RoutingRunStore` correct when the run-ops layer routes across
more than two Postgres stores. Today it routes between a gen-1 `new`
dedicated database and a `legacy` control-plane database; this
generalizes every routing policy to N shards while keeping the two-store
behaviour byte-identical.

The change sets the four routing decisions that were implicit in code
order, and fixes one hazard that failed silently:

- **Id → shard key.** The router resolves a shard key with
`resolveShard` instead of the binary residency classifier, so a gen-2 id
reaches its own shard through the keyed map.
- **Membership vs routing.** `#distinctStores` (one entry per physical
database, aliases excluded by a declared `aliasOf`) drives every sum,
probe, and merge; `#shards` drives routing. An aliased shard can no
longer make a sum count one database twice.
- **Probe order.** A keyless lookup stays a sequential short-circuit at
two stores; above two it fans out in parallel, picks by precedence,
tolerates a single down leg, and keeps the canonical not-found throw on
the legacy leg.
- **Precedence and duplicates.** One merge helper across all four merge
sites. A duplicate id confined to `{new, legacy}` stays silent (the
known drain-mirror case); any other cross-shard duplicate increments
`runops_shard_duplicate_id_total` and logs at error level.
- **Disjoint sum (the silent hazard).** `countPendingWaitpoints` and the
waitpoint collector now partition absent ids by shard and **union by
id** rather than summing counts. A drain-mirrored waitpoint on both
gen-1 stores is counted once, so a blocked run can no longer hang
forever on a double-counted pending waitpoint.
- **Waitpoint completion.** A gen-2 waitpoint completes on its own
shard, overriding the legacy pins; a cuid waitpoint keeps its two-member
gen-1-pair probe unchanged.
- **Fail-loud creates.** A create with no shard key throws instead of
silently defaulting to `new`. An id resolving to an unconfigured shard
throws instead of being dropped.

Two new counters are exported: `runops_shard_duplicate_id_total` and
`runops_waitpoint_probe_fallback_total`.

## Why it is safe to merge

With only `{new, legacy}` configured every generalized rule reduces to
today's behaviour. `resolveShard` returns exactly what the old
classifier returned for every id shape that exists today, and no gen-2
id is minted yet. The only intentional behaviour change is the fail-loud
create throw; an enumeration of production call sites confirmed no
caller trips it.

## Testing

- New container-free algebra suite (50 cases) over probe order,
precedence, the duplicate alarm, the disjoint-sum partition, the
waitpoint probes, and the fail-loud paths.
- New `runOpsStore.nShardMatrix.test.ts` runs a four-store matrix
(legacy + new + two gen-2 shards) against real Postgres containers: the
disjoint-sum union, the alias topology, cross-tree completion,
pagination merges, and mixed-id hydration.
- New `makeNShardRunOpsPostgresTest(k)` fixture in
`@internal/testcontainers`.
- Full run-store corpus green: 71 files, 480 tests. Typecheck, lint,
format, and knip all clean.

## Notes

- Draft: opened for review; not marking ready yet.
- No changeset or `.server-changes` file: internal routing
infrastructure, no user-visible behaviour change.
- TRI-13427.

Auto-scroll now only follows while you are at the bottom. Scrolling up
pauses it; scrolling back to the bottom, or clicking the new
scroll-to-bottom button in the log header, resumes it. When you are at
the bottom the same button scrolls to the top. Switching to another
deployment starts at the bottom again.

…4777)

The environment variable key and value inputs did not set an
autocomplete attribute, so browsers could offer to autofill or save
typed values as saved credentials. This sets `autoComplete="off"` on
those inputs in both the create and edit forms, matching the
`autoComplete="off"` convention already used on the other
credential-name inputs.

`autoComplete="off"` is a best-effort hint. Browsers may still ignore it
for password-typed fields, so this is defense-in-depth hardening, not a
hard guarantee that a password manager cannot store the value.

…4764)

Part of the RunOps N-way sharding work.

This lets the webapp hold N run-ops stores, configured by a single
`RUN_OPS_SHARDS` JSON descriptor, and routes to them through the
existing keyed router. **Inert with `RUN_OPS_SHARDS` unset** — the
topology, the wiring and `ROUTING_ENABLED` are byte-identical to today.

## What's here

- **`RUN_OPS_SHARDS`** — a zod-validated JSON array of shard descriptors
(`key`, `region`, `url`, `replicaUrl`, `directUrl`, `replication`,
`knobs`, `aliasOf`), validated at boot in the `parseMachinePresetCsv`
style. Unset or `[]` → no shards.
- **One run-ops client factory** —
`buildRunOpsWriterClient`/`buildRunOpsReplicaClient` collapse into one
`buildRunOpsClient` parameterized by role and resolved pool knobs. The
control-plane builders (`buildWriterClient`/`buildReplicaClient`) are a
separate path and stay untouched; every resolved value matches the
former builders.
- **Shard loop in `selectRunOpsTopology`** — one client pair per
descriptor; an `aliasOf: "new"` descriptor reuses the new store's
clients by reference and opens no pool.
- **N-way `buildRunStore`** — builds N dedicated stores + the keyed
router via a new `RoutingRunStore.fromShards`, keeping the two-store
compat router when no shards are configured.
- **`UnknownShardKey`** — raised when an id resolves to an unconfigured
key; never falls back to another store. `fromShards` injects
`resolveShard` so a gen-2 id routes to its own shard.
- **Per-shard transaction resilience** — each shard gets its own retry
budget.
- **Mint bound** — `computeMintShard` intersects the active mint list
with the configured descriptor keys, so a key with no descriptor is
never minted into.
- **Boot table** — logs `key`, address fingerprint (host:port/db, no
credentials), and role, only when shards are configured.

## Ordering constraint

Do **not** configure a `RUN_OPS_SHARDS` descriptor in any environment
until the routing-semantics change (TRI-13427) lands — three fan-out
sites still truncate at N>2. Merging this PR alone is safe (inert with
the var unset); configuring a descriptor is what must wait.

## Testing

- Run-store corpus: green with zero test-file diffs (the bit-identical
proof for the compat router).
- `runOpsDbTopology.test.ts` 17/17, `runStore.server.test.ts` 4/4,
`runOpsMigration` family 149/149.
- New unit suites: descriptor validation, pool-knob value tables,
`fromShards` routing + `UnknownShardKey`, boot-table formatter, mint
bound.
- typecheck (webapp + run-store), knip, lint, format: pass.

## Changelog

Internal run-ops sharding infrastructure. No changeset or
`.server-changes`: the change is inert with `RUN_OPS_SHARDS` unset and
has no user-visible behaviour.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Deployments currently leave little analytical trace. This PR makes every
deployment emit two analytics events to enable useful queries. It also
enables comparing deployments across build paths, CLI versions,
runtimes, and orgs.

### Where the events come from

```
 trigger deploy
      │
      ▼
  initialize ─────────────────────────────▶ ✨ deployment.initialized
      │ createdAt
      ▼
   PENDING      waiting for a build slot        ┐
      │ startedAt                               │ queue time
      ▼                                         ┘
  INSTALLING    build server installs deps      ┐
      │ installedAt      (native paths only)    │ install time
      ▼                                         ┘
   BUILDING     the image is built              ┐
      │ builtAt                                 │ building time
      ▼                                         ┘
  DEPLOYING     indexing + registry push        ┐
      │ deployedAt / failedAt / canceledAt      │ deploying time
      ▼                                         ┘
  DEPLOYED · FAILED · TIMED_OUT · CANCELED
      │
      └───────────────────────────────────▶ ✨ deployment.finished
```

`deployment.finished` fires exactly once, whichever way the deployment
ends, and is backdated to cover the deployment's real lifetime. Not
every path visits every state (Depot deploys skip PENDING/INSTALLING,
for example) — a phase duration is simply omitted when its state was
never entered.

### What each event carries

- **Which path built it**: `depot`, `native`, or `native_local_bundle`
- **How it ended**: status, plus an error class and message when it
failed
- **How long each phase took**: queue, install, building, deploying, and
total — derived from the timestamps above
- **Who and with what**: org, project, environment, runtime, CLI
version, and how the deploy was triggered (CLI, GitHub, Vercel)

With that, one query gives failure rate per build path, duration
percentiles per phase, adoption per CLI version, or a per-org health
table.

### Fixes that ride along

- The old `deployment.outcome` span was silently dropped ~95% of the
time (it was subject to trace sampling). The new events opt out of
sampling explicitly, so every deployment is counted.
- The fail/timeout/finalize transitions were racy: a late timeout could
overwrite a successful deployment. They now use guarded writes, so
exactly one caller wins the terminal transition — and exactly one event
is emitted.
- Canceled deployments previously recorded nothing; they do now.
- The deployment's CLI version is now stored at initialization (new
nullable column), so even deploys that fail early are attributable to a
CLI release.
- Telemetry is flushed on shutdown (the last batch used to be lost on
every webapp deploy), and an optional second exporter can mirror just
these events into a dedicated dataset.

…vents (#4785)

Adds `$trigger.org.slug` and `$trigger.project.name` attributes to the
`deployment.finished` / `deployment.initialized` events (follow-up to
#4778).

…bases (#4780)

## Summary

The run-ops boot interlocks and the migration entrypoint each assume
exactly two run-ops
databases. This generalizes them to any number, so a deployment that
configures
`RUN_OPS_SHARDS` gets the same safety guarantees it gets today with two
stores: no two stores
may point at one database, every store that owns its own database must
replicate to
ClickHouse, and every store must have its schema migrated.

With `RUN_OPS_SHARDS` unset, nothing changes. The distinctness check
over a two-element set is
the pairwise compare it replaces, replication coverage is the check it
was, and the entrypoint
runs the same two migration invocations.

A shard may declare `aliasOf: "new"`, which shares an existing store's
client by reference. An
aliased shard is not its own database, so it is exempt from the
distinctness check and needs no
replication slot of its own. Every check keys that exemption on the
declared field, never on
client object identity: two client objects can sit over one database,
which identity comparison
cannot see.

## Design

**Distinctness.** `probeDistinctDatabases` compared two URLs. It now
delegates to
`probeDistinctStores`, which reads every fingerprint in parallel and
groups them by system
identifier and database name. Any two stores under one key refuse the
boot. The old pairwise
entry point stays, so its existing container tests are the proof that
set uniqueness over one
pair gives the verdict it gave before. Fail-closed is unchanged: a probe
that cannot answer
returns not-distinct, because "distinct" is a positive claim a failed
probe cannot support.

**Co-residency.** The advisory runs once per store against the control
plane. The legacy
emission keeps its exact call shape and its untagged metric series, so
an existing dashboard
does not change. Each shard emits its own point carrying its shard key.
Every store emits
before any enforcement throw, so one offending store never costs another
store its metric.

**Replication.** `buildReplicationSources` appends one source per shard
that owns its own
database, taking the slot, publication and origin generation its
descriptor declares.
`assertReplicationCoversSplit` then requires a source per such shard.

That check also closes a hole it inherited. The descriptor parser
validates uniqueness among
shards only, so a shard could take the slot name, publication name or
origin generation of the
legacy or the new source. The replication service does validate this,
but it throws from its
constructor, and the caller reaches that constructor only after shutting
the bootstrap instance
down:

```ts
if (sources.length > 1) {
  await service.shutdown();                       // legacy stream stops here
  service = new RunsReplicationService({ ... });   // throws: duplicate slotName
}
```

The throw was not a `SplitReplicationMisconfiguredError`, so the process
stayed up with no
replication at all, legacy included, behind one logged line. That is the
silent ClickHouse
under-count the error exists to prevent. The check now runs at the boot
gate, before anything is
torn down, and raises a subclass the existing exit path already
recognizes. A correct deployment
already satisfies it, because two consumers on one WAL slot is a data
race that cannot work.

**Migrations.** Every shard runs the identical schema, so a new shard is
the existing migrations
against a new DSN. The runner image has no `jq`, so a small node script
prints one DSN per line
and the entrypoint loops over them. The loop is a `for` and not a `while
read` pipeline: a
pipeline subshell swallows a failed migration on any iteration but the
last, which would let a
broken shard boot. Tracing stays off across the capture and the loop,
because `set -x` prints an
assignment and a DSN carries credentials.

Verified end to end against real Postgres containers for the fingerprint
probes, and against the
real shell block with a stubbed migration command: an aliased shard is
skipped, `directUrl` wins
over `url`, a failing shard stops the container on the first failure,
and a malformed descriptor
stops it before it migrates anything.

Stacked on #4764.

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…off-by-default dial (#4765)

## Summary

Adds a `RunStore` decorator that mirrors execution snapshots into Redis
alongside Postgres, plus the orphan-key sweep and the fault-injection
suite that prove the write protocol converges after a crash. Nothing
constructs it, so merging this changes no behaviour: the configuration,
the production wiring and the Redis client all arrive in later work.

The execution-state log is the hottest table in the run graph, and
moving it out of Postgres has to happen without a big-bang cutover. This
is the attachment point for that: a decorator that wraps the existing
storage interface and intercepts only the methods that touch snapshots,
so none of the many callers change.

## Design

Write order is the correctness property, and the two orders differ on
purpose.

A transition writes Postgres first and Redis second. A crash in the gap
leaves a run whose latest snapshot is stale, which is the state the
heartbeat stall watchdog already heals in production today.

A birth writes Redis first and Postgres second. A crash there leaves an
unreachable key for a run that does not exist. Postgres first would
instead leave a run with no snapshot at all, which the engine treats as
a hard error, so the run would be stuck.

Each order is chosen so the state a crash leaves behind is the harmless
one. A lost cross-store write is never recovered by a transaction or an
outbox; recovery is always the existing stall and repair job. A failed
append retries, then hands the run to that job, and never rethrows,
because Postgres has already committed and a throw would turn a healable
gap into a caller-visible error.

Inside a transaction the Redis half is staged and flushed only after the
commit, so a rollback cannot leave Redis holding a transition that never
happened.

Reads are shape matched. Two of the snapshot reads take arbitrary Prisma
arguments, and a key-value store cannot answer an arbitrary query, so
the decorator recognises exactly the shapes the engine sends and
delegates everything else. A miss falls back to Postgres, which is also
how runs created before any cutover keep working.

The sweep reaps under two rules, because neither can see what the other
leaves behind. A finished run whose keyspace never received its
completion expiry gets one applied. A keyspace with no run row at all,
past an age threshold, is deleted; that is a crashed birth, which is
non-terminal so it carries no expiry and has no run row, so the first
rule can never match it.

## Inertness

Three independent reasons this is a no-op if merged alone:

- Nothing constructs the decorator or the Redis store outside tests.
- No configuration reaches it, so the dial stays at its off position,
which is a pass-through that makes no Redis call.
- The existing Postgres store gains an off-by-default flag and two
optional input fields. Both default to today's behaviour, and only the
decorator would ever supply them.

## Notes for review

The snapshot id and the creation instant are both minted by the
decorator and written into both stores, so one snapshot has one identity
and one timestamp wherever it is read. Without that, the two stores
disagree on values that later tooling has to compare, and the cursor for
a snapshot window resolved from one store misfilters the window walked
in the other.

Three defects in this work passed the full existing test suites before
being found by review rather than by a test: the decorator wrote no wait
cycle at all, the snapshot window dropped the ordering used to give each
completed waitpoint its position in a batch, and the two stores stamped
different creation times. The common cause was that no test drove a
snapshot that actually carried waitpoints, and that the parity suite
compared a timestamp against a value it had just read back from the row
it was checking. Both gaps now have tests.

…code, and follow-ups (#4784)

Three bugs on the project integrations page, one commit each for the two
reported ones and four for the follow-ups found while fixing them.

## `chore`: remove unreachable code on the integrations page (TRI-12645)

Two notification panels in `VercelSettingsPanel` could never render:

1. The **"Failed to load Vercel settings"** panel was gated on a
`hasError` state whose setter is never called anywhere, so it was
permanently `false`.
2. The **"connection expired"** banner *inside* the `connectedProject`
branch was unreachable: `VercelSettingsPresenter` only populates
`connectedProject` on its success exit, which hardcodes `authInvalid:
false`, while both `authInvalid: true` exits return `connectedProject:
undefined`.

Removing them makes the surrounding `!showAuthInvalid` guards vacuous,
and the `onboardingData?.authInvalid` disjunct redundant — the loader
already folds onboarding auth state into `authInvalid` before it reaches
the component.

**No behaviour change.** An org with a connected project and an expired
token still gets the banner, from the branch below (untouched).

## `fix`: gate Staging settings on plans without a Staging environment
(TRI-12646)

The ticket's premise was inverted, and I've corrected it there. In Git
settings, **Preview** is the row that's correctly gated; **Staging** is
the one with no gate at all:

- Preview swaps its switch for an Upgrade button, and
`projectSettings.server.ts` neutralises a forged
`previewDeploymentsEnabled=on`.
- Staging was a plain always-editable `Input`, and
`validateStagingBranch` only checked the branch existed on GitHub. An
org without a staging environment could type a tracking branch, hit
Save, get a success toast, and have it silently do nothing.

Staging and Preview environments are created together for projects on a
plan that includes them, so gating one and not the other was an
oversight.

The Staging row now mirrors the Preview row. Server-side it ignores the
submitted branch when there's no staging environment, but **preserves
the stored branch rather than clearing it** — deliberately different
from the Preview handling. Forcing a boolean off is harmless; forcing a
*string* off would wipe a tracking branch the org had already configured
the first time they saved after losing the environment.

The Vercel write path had the same gap: `update-config` /
`complete-onboarding` / `update-env-mapping` never re-derived available
env slugs server-side, so `["stg","preview"]` could be persisted for a
project with neither environment, and
`createDefaultVercelIntegrationData` turned preview on unconditionally.
Both now filter against the project's actual environments, via a pure
`restrictConfigToAvailableEnvSlugs` helper that only touches keys
present on the input.

## `fix`: show build settings when the GitHub app is disabled
(TRI-13488)

The page wrapped Git settings, the Vercel section **and** build settings
in one `githubAppEnabled` guard, so with the GitHub app off it rendered
an empty container.

The Vercel section genuinely depends on GitHub — it can't sync
environment variables or link deployments without a connected repo — so
it stays gated. Build settings don't: they also apply to CLI deploys run
with `--native-build-server`, exactly as the section's own description
states. They now render regardless.

## `fix`: stop the Vercel onboarding modal spinning forever (TRI-13488)

`computeInitialState` starts in `loading-projects` whenever the org has
a Vercel integration but no onboarding data yet, and the effect that
escapes it waits for `availableProjects !== undefined`. When
`getOnboardingData` returns `null` — it does that on any thrown error,
and when the org integration row is missing — nothing ever arrives.

The empty-array case self-resolves (`[] !== undefined`), so this is
specifically the null case. The route can tell "still loading" from
"loaded nothing" because its fetcher always requests
`?vercelOnboarding=true`; it now passes that down and the modal explains
the failure with a retry and a link to check the integration's access on
Vercel.

## `fix`: match staging and preview environments consistently
(TRI-13488)

The four places that ask "does this project have a staging / preview
environment?" disagreed. `VercelSettingsPresenter` matched on type with
no parent filter, so any preview *branch* row satisfied it — branches
are `PREVIEW` rows too. `GitHubSettingsPresenter` and
`ProjectSettingsService` matched on slug instead.

Slug is the weaker key: it's derived at creation time and legacy rows
can carry something else, which is why
`memberDevelopmentEnvironmentWhere` deliberately avoids it. All four now
match on `type` plus `parentEnvironmentId: null`, which excludes
branches without depending on the slug being canonical.

## `fix`: explain when no Vercel environment can be mapped to Staging
(TRI-13488)

Reported while reviewing the branch. The Staging build settings show
*"Set a Vercel environment for Staging first."* whenever the project has
a staging environment and no mapping — but the control that sets the
mapping only rendered when the Vercel project had at least one custom
environment:

```
hint:     hasStagingEnvironment && !configValues.vercelStagingEnvironment
control:  hasStagingEnvironment && customEnvironments.length > 0
```

So a Vercel project with no custom environments, or one whose custom
environments failed to fetch (the presenter swallows that error to
`[]`), got an instruction with nothing to act on. Both conditions
predate this PR.

The mapping row now always renders alongside the hint and explains what
to do when there's nothing to choose from, and the build-settings hint
says the same thing.

## `chore`: remove the remaining dead code (TRI-13488)

- The `"installing"` `OnboardingState` is unproducible — no `setState`
call yields it — so its redirect effect, switch arm, `isLoadingState`
conjunct and the `vercelAppInstallPath` import it was the only user of
are all dead.
- `(state as string) !== "completed"` sits in a branch where TypeScript
has already narrowed `"completed"` out; the cast is what let it compile.
- `hideSectionToggles` was only ever passed alongside
`layout="settings"` but only read inside `layout="card"` blocks, so it
could never take effect. Removed the prop entirely.
- Unused bindings and the helpers only they referenced: `envSlugLabel`,
`_formatSelectedEnvs`, `_CompleteOnboardingForm`,
`_handleFinishOnboarding`, and the rest.

No behaviour change in that commit.

## Not included

The three overlapping modal-open effects in
`settings.integrations/route.tsx` are left alone — they're defensive
against a close-then-reopen race, and untangling them is a behavioural
risk with no user-visible payoff.

## Verification

`pnpm run typecheck --filter webapp`, `pnpm run lint` and `pnpm run
knip` are clean. New `apps/webapp/test/vercelIntegrationConfig.test.ts`
covers the slug restriction and the default-config seeding (both pure
functions); 39 tests pass across it and the three existing
Vercel/project-settings files.

The new `projectId` + `slug` query is served by the existing
`@@unique([projectId, slug, orgMemberId])` prefix — same access pattern
as the preview check it mirrors.

refs TRI-12645, TRI-12646, TRI-13488

…ncy (#4781)

Gives read-through and idempotency their gen-2 shard arms, so an id that
names its own shard is read there and nowhere else.

#4764 has landed, so this now targets `main` directly and no longer
depends on an unmerged branch. It builds on what that PR supplied:
`resolveShard`, `runOpsShardHandles` and the keyed router.

TRI-13431

## What changes

**Read-through routes by `resolveShard`, not by the binary residency
classifier.** A gen-2 id reads its own shard's replica once and probes
no other store. A gen-1 v1 id still reads new only.

**Callers now declare `idKind`.** A cuid gives no way to tell a run id
from a waitpoint id, and the two must route differently:

- a legacy-classified **run** id reads the legacy replica only — there
is no cuid run migration, so the new-store probe cannot find it;
- a cuid **waitpoint** keeps the new-first pair probe, which is
load-bearing because a cuid waitpoint can be co-located with its run on
the new store.

There is no default, because a default would pick one of those arms
silently. The field `runId` is renamed to `id`, since it carried both
kinds already.

**`ReadThroughResult` carries `found`.** `source` is an open-ended union
once shards exist, so a consumer testing found-ness by listing the hit
sources reads a gen-2 hit as a miss. One consumer did exactly that.
Discriminating on `found` makes that class of bug a compile error rather
than something a reviewer has to spot.

**Idempotency resolves its client through one shard-keyed map.** Both
call sites go through `clientForShardKey`, so they cannot disagree about
which store owns an id. An absent key takes an explicit logged branch to
the fallback, not a silent legacy default. The `classify` seam is
retyped to return a `ShardKey`: `Residency` (`"NEW"`) and the reserved
shard keys (`"new"`) differ only by case, and `ShardKey` collapses to
`string`, so the compiler would not have caught feeding one into the
other.

The dead `isMigrated` branch is deleted. Nothing implemented it, and the
one production comment recorded that omitting it was deliberate.

**`PostgresRunStore._residency` widens to `ShardKey`.** Still unused;
the store stays unaware of its siblings.

## Two behaviour fixes found while doing the above

**An unconfigured shard key logs and returns not-found instead of
throwing.** The waitpoint route takes the id from a URL parameter, and
any base32hex core plus `[a-z0-9]` plus `"2"` parses as gen-2. The route
turns a throw into a 500, so throwing here would let any authenticated
client generate 500s and error logs by guessing shard chars, of which
there are 36. An error-logged not-found is neither silent nor a
misroute. Throwing stays correct on the router path, where ids are
minted rather than received.

**The two cross-seam batch hydration sites were gen-2 blind.**
`hydrateRunsAcrossSeam` and `ApiBatchResultsPresenter` classified with
the binary `ownerEngine`, so a gen-2 run id joined the gen-1 `new`
group, missed there, and — classifying dedicated-family — never reached
the legacy probe either. The id was dropped from a bulk-action page and
from batch results with no error. Both now partition ids by shard key
and read each configured shard once.

Also: a gen-2 waitpoint that missed its shard replica fell back to the
gen-1 new writer, a different database, silently disabling
read-your-writes for the freshly minted token that fallback exists to
serve. It now falls back to its own shard's writer.

## Merge safety

Inert while `RUN_OPS_SHARDS` is unset: the shard maps are empty, so
every gen-2 arm is unreachable, and gen-2 minting is not live yet.

The one live change is the gen-1 run arm, and it removes work rather
than adding it. `RoutingRunStore.findRun` never forwards the caller's
client object — it routes by id and reads only the client's presence and
replica brand — so `readRunForEvent`'s "new" closure already resolved a
legacy-classified run id to the legacy store. The arm removes a
duplicated read of the legacy replica. A test pins this, because a
future caller passing a raw client and a run id would lose the
pre-cutover 27-char case, which is new-resident but classifies legacy.

## Testing

14 tests added, testcontainers throughout, no mocks. 22 affected test
files pass; typecheck, lint, format and knip are clean.

Both arms were verified by neutralising them and confirming the new
tests fail. The batch-results test needed rewriting after that check:
the first version passed with the fix neutralised, because it used one
container as both the gen-1 new client and the shard replica, so it was
not testing what it claimed.

Note for review: run testcontainer suites in small batches. Sixteen at
once starves Docker and everything times out at 60 seconds.

The run-ops legacy-guard baseline is refreshed in its own commit. The
baseline is keyed by line number, so partitioning the batch-results read
shifted four pre-existing entries and added one. Baselined violations in
that file go from four to five, all reads; the new one is the shard read
beside two gen-1 reads already there.

No changeset and no `.server-changes` entry: a user notices nothing
while the flag is unset.

## ✅ Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

---

## Testing

Reproduced with `useTriggerChatTransport` + `useChat` and the stop
pattern from the ai-chat frontend docs:

1. Send a message so a turn is streaming.
2. Call `transport.stopGeneration(chatId)`, then `useChat`'s `stop()`.
3. Send another message.

Before this change the second turn never renders: no parts arrive,
`status` stays `streaming`, and the session stays `isStreaming: true`,
so a stop button stays on screen until the page is reloaded. The run
itself is fine and everything persists, so a reload shows the full
response.

Cause: `stopGeneration` sets `state.skipToTurnComplete = true`, and the
read loop only clears that when it sees a `TURN_COMPLETE` record. The
abort closes the reader before that record arrives, so the flag survives
into the next turn and every record of that turn is skipped, including
its own `TURN_COMPLETE`.

After this change the same sequence streams the second turn normally.
Verified against 4.5.11 and 4.5.12 (both affected) with the equivalent
patch applied to the built SDK.

---

## Changelog

Reset `skipToTurnComplete` when a new chat turn or action is sent, so a
message sent after `stopGeneration` streams normally instead of leaving
the chat stuck in a streaming state.

---------

Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: nicktrn <55853254+nicktrn@users.noreply.github.com>

#4644)

## Summary

Adds `chat.messages.hasPending()` and `chat.messages.next()` so a custom
agent
loop can inspect pending chat input without consuming it and take one
record at a
time, and fixes four ways a chat could mishandle input across a restart:
a
message silently lost, a recovered answer cut off by a stop the user had
already
pressed, a retried send answered twice, and a record the agent had no
consumer
for blocking every message queued behind it.

```ts
if (await chat.messages.hasPending()) {
  const record = await chat.messages.next({ timeoutInSeconds: 0 });
  if (record) handle(record.payload);
}
```

## Why the fixes came together

`session.in` carries records for consumers whose delivery needs differ.
A user
message must be delivered eventually, so it can wait arbitrarily long
for a turn
to take it. A stop only means anything to the turn that is live when it
lands.
Progress along the channel was tracked as one sequence number, and one
number
cannot say "control applied through 7, message 3 still owed" at the same
time.
Each of the bugs above is that mismatch surfacing somewhere different.

So instead of a rule per symptom, records are now classified once and
handed to
one route, and each route declares two things: whether it holds a record
when no
consumer is ready, and whether a record it never handled has to survive
into the
next boot. The resume cursor, the replay window and the
discard-the-unowned
behaviour are then derived from route state rather than maintained
beside it, and
`hasPending()` answers from the message queue instead of the head of a
buffer
shared with every other kind.

The wire is unchanged. Both cursors on the turn boundary keep their
meanings, so
existing chats resume as before and there is no webapp change.

## Behaviour worth calling out

`chat.writeTurnComplete()`'s `sessionInEventId` is the cursor that is
safe to
resume from, not the sequence of the record the turn answered. It is
held back
behind any message still waiting to be handled, so a value below the
record you
just handled is expected rather than a sign of a lost turn.

The stop fix also covers chats whose most recent turn was completed by
an older
SDK, by resolving the replay window from the channel when the boundary
does not
carry one. The trade there is deliberate: a stop that landed in the
moments
before boot and was never applied is dropped along with the replayed
ones,
because a stop the user can press again beats a stale one killing an
answer they
are waiting for.

## Verification

Thirteen reproductions against a local stack, each driving real runs
rather than
mocks, covering the documented `next()`/`hasPending()` loop, suspend and
resume, a
crash between consuming a message and writing turn-complete, a retried
send whose
idempotency claim is lost, and a continuation boot that must not replay
answered
messages. Where applicable each was also run against `main`, so the
fixes are
differences rather than assertions.

Five further legs on a deployed environment, which the earlier revisions
of this
branch did not cover at all: a message appended while the run is
genuinely
checkpointed, a message appended while the run is dead, the
stop-after-crash case
on the real crash path, and both version-skew directions (a newer worker
resuming
an older worker's turn boundary, and an older worker resuming a newer
one's).

Two of those restart fixes also have a browser-driven red and green pair
on a
deployed environment, staged identically on both sides and differing
only in the
SDK. For the lost-message fix, the unanswered message is replayed and
answered in
full here, and is never replayed at all on the released SDK. For the
stop fix,
both sides replay the message and diverge on the stop itself: it is
declined here
and the answer completes, while the released SDK re-applies it and the
recovered
answer dies before it streams.

The routing decision itself is a pure state machine, so it also has a
property
test over every interleaving of the record kinds crossed with each crash
point,
checked by mutation to confirm it fails when the cursor arithmetic or
the replay
window is broken.

## Known and not addressed here

The read of the woken record is unbounded, so a wake with nothing to
read makes
`wait()` outlive its own waitpoint. Tested and not a deadlock, since the
read
defers to the next record, but bounding it is a separate change with its
own
test.

Separately, and not caused by this branch: a run that crashes while a
message is
still queued is not replaced until the next inbound append, so that
message waits
rather than being recovered on its own. Worth its own issue.

Also not caused by this branch, but worth knowing when reading the
release note: a
chat page that stayed open across the crash keeps showing the partial
answer it
already received, so the recovered answer only appears after a reload.
The answer
itself is persisted correctly. The gap is on the client, which does not
apply a
re-delivered turn over a partial it already holds.

---------

Co-authored-by: Eric Allam <eric@trigger.dev>
Co-authored-by: Eric Allam <eallam@icloud.com>

## Summary

Reloading a browser chat mid-turn can replay a completion event for an
older input and close the active turn too early.

This persists the last browser-owned input sequence and reuses it on
reconnect, so older completion events are ignored. The sequence is
cleared after the matching boundary, and reconnect avoids the
settled-peek shortcut while that sequence is active.

The persisted field is optional, so sessions without it keep their
existing behavior.

## Testing

- `pnpm --dir packages/trigger-sdk run test ./src/v3/chat.test.ts
./test/chat-turn-correlation.test.ts --run` — 67 passed
- `pnpm --dir packages/trigger-sdk run test --run` — 32 files, 379 tests
passed
- `pnpm run build --filter @trigger.dev/sdk`
- `pnpm run format`
- `pnpm run lint`

## Changelog

Browser chats now keep the active turn open across page reloads when
older completion records are replayed.

## ✅ Checklist

- [x] I have followed every step in the [contributing
guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md)
- [x] The PR title follows the convention.
- [x] I ran and tested the code works

💯

Co-authored-by: Matt Aitken <matt@mattaitken.com>

## Summary
3 improvements.

## Improvements
- Show how to override the default cron window in schedule policy
warnings.
([`630eb75d1`](630eb75))
- Thrown error `cause` chains are now captured and shown. When a task
throws an error that wraps another one, the run's error in the
dashboard, the CLI dev output, and failure alerts all carry the chain
instead of only the outermost message.
([`4f36f614b`](4f36f61))
  
  ```ts
throw new Error("Could not sync the customer", { cause: originalError
});
  ```
  
The chain is flattened outermost first, capped at five causes, and cycle
safe. It also rides on the `error` of API and realtime run records as a
`causes` array, and `triggerAndWait` and `triggerAndSubscribe` rebuild
it as a native `cause` on the error they hand back, so `err.cause` works
in your own catch blocks.
- Keep summarized assistant steps and tool results out of future
`chat.agent()` model context after inner compaction, while preserving
the full visible conversation.
([`fc77bfc9a`](fc77bfc))

<details>
<summary>Raw changeset output</summary>

# Releases
## @trigger.dev/build@4.6.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.6.3`
## trigger.dev@4.6.3

### Patch Changes

- Show how to override the default cron window in schedule policy
warnings.
([`630eb75d1`](630eb75))
- Thrown error `cause` chains are now captured and shown. When a task
throws an error that wraps another one, the run's error in the
dashboard, the CLI dev output, and failure alerts all carry the chain
instead of only the outermost message.
([`4f36f614b`](4f36f61))

  ```ts
throw new Error("Could not sync the customer", { cause: originalError
});
  ```

The chain is flattened outermost first, capped at five causes, and cycle
safe. It also rides on the `error` of API and realtime run records as a
`causes` array, and `triggerAndWait` and `triggerAndSubscribe` rebuild
it as a native `cause` on the error they hand back, so `err.cause` works
in your own catch blocks.

- Updated dependencies:
  - `@trigger.dev/core@4.6.3`
  - `@trigger.dev/build@4.6.3`
  - `@trigger.dev/schema-to-json@4.6.3`
## @trigger.dev/core@4.6.3

### Patch Changes

- Thrown error `cause` chains are now captured and shown. When a task
throws an error that wraps another one, the run's error in the
dashboard, the CLI dev output, and failure alerts all carry the chain
instead of only the outermost message.
([`4f36f614b`](4f36f61))

  ```ts
throw new Error("Could not sync the customer", { cause: originalError
});
  ```

The chain is flattened outermost first, capped at five causes, and cycle
safe. It also rides on the `error` of API and realtime run records as a
`causes` array, and `triggerAndWait` and `triggerAndSubscribe` rebuild
it as a native `cause` on the error they hand back, so `err.cause` works
in your own catch blocks.
## @trigger.dev/python@4.6.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/sdk@4.6.3`
  - `@trigger.dev/core@4.6.3`
  - `@trigger.dev/build@4.6.3`
## @trigger.dev/react-hooks@4.6.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.6.3`
## @trigger.dev/redis-worker@4.6.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.6.3`
## @trigger.dev/rsc@4.6.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.6.3`
## @trigger.dev/schema-to-json@4.6.3

### Patch Changes

- Updated dependencies:
  - `@trigger.dev/core@4.6.3`
## @trigger.dev/sdk@4.6.3

### Patch Changes

- Thrown error `cause` chains are now captured and shown. When a task
throws an error that wraps another one, the run's error in the
dashboard, the CLI dev output, and failure alerts all carry the chain
instead of only the outermost message.
([`4f36f614b`](4f36f61))

  ```ts
throw new Error("Could not sync the customer", { cause: originalError
});
  ```

The chain is flattened outermost first, capped at five causes, and cycle
safe. It also rides on the `error` of API and realtime run records as a
`causes` array, and `triggerAndWait` and `triggerAndSubscribe` rebuild
it as a native `cause` on the error they hand back, so `err.cause` works
in your own catch blocks.

- Keep summarized assistant steps and tool results out of future
`chat.agent()` model context after inner compaction, while preserving
the full visible conversation.
([`fc77bfc9a`](fc77bfc))
- Updated dependencies:
  - `@trigger.dev/core@4.6.3`

</details>

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Adds support for environment variables with empty-string values, rolling
out gradually. Once enabled for your organization, an empty value stays
stored and available to tasks, including through environment-variable
imports and Vercel sync. Deleting a variable remains a separate action.
Also fixes local dev environment precedence so freshly resolved project
values reach task processes correctly.

Mono-RevId: 25dea78a2c72a984a9bd9ac432c11026f220576c

Use Node.js 24 for dashboard agent task deployments.

Mono-RevId: a0d27501dc1ede88a89955bed4dc2f346bed062f

Mono-RevId: e42ec673635842461d2261b89345b9a2cc847db4

Add opt-in preview branch auto-archiving after a configurable period
without deployments. Projects can protect named branches and preview the
effect of their settings before saving. Availability is controlled by an
organization or global feature flag and is disabled by default.

Mono-RevId: 96ef71472cc849ed8fffbef47fc4503987ba6288

Each server process opened Redis pub/sub subscriptions for every
socket.io namespace at startup, including namespaces it would never
serve a connection on. Every broadcast was therefore delivered to every
process, and all but the few holding a relevant connection discarded it.

Subscriptions are now opened per namespace the first time that process
serves a connection on it, which keeps broadcast fan-out to the
processes that can act on it. Behaviour is unchanged for a single
process deployment, where the first worker or dev worker connection
opens the same subscriptions as before.

Mono-RevId: c14a7b60cd4303f463a24c6e044b4d48cc25e981

Balance internal test files across packages and split the workload
across two runners to reduce CI waits. Preserve package test
configuration, refresh duration estimates, and combine all shard reports
for timing updates.

Mono-RevId: f43f740e7a8db6a812a157a8a02b2a15a7ac66fa

When the server returns `200` without records, chat subscriptions can
retry indefinitely. The client stall timer fires after 60 seconds,
before S2 closes the response at its 120-second timeout, so the existing
EOF budget never applies. Terminal failures also leave persisted
`isStreaming` state active, so a reload starts another subscription.

Normal chat subscriptions now permit five reconnects after stall
timeouts. A separate stall counter preserves unlimited retries for
retryable connection failures, fetch timeouts, and browser wakeups.
Decoded records restore the stall budget. Terminal failures clear state
only for the owning subscription, and watch subscriptions remain
unlimited.

The existing stall timer already ignores keepalives because the parser
drops them before the timer reset. The comment correction does not
change that behavior.

## Checklist

- [x] The PR title follows the contribution convention.
- [x] The changes include tests and a changeset.
- [x] Local package builds, formatting, lint, and knip passed.
- [ ] Maintainer CI and reference-project validation.

## Testing

- Regression tests reproduce timeout exhaustion, indefinite stalls,
stale terminal state, and recovery beyond five connection failures.
- Full suites at `8149b96`: 695 SDK tests and 1,122 core tests passed.
- Final error-message and constructor changes: all 43 stream tests
passed.
- Core and SDK builds passed.
- Repository formatting, lint, and knip passed.
- General and security review passes found no remaining defects.
- The debug-marker check reports existing markers in
`apps/webapp/app/services/previewAutoArchive.server.ts`; the changed
files contain none.

The tests use local HTTP servers. They cover separate stall limits,
fetch timeouts, body failures, keepalives, progress resets, mixed
failures, wakeups, cancellation, watch recovery, and replacement
ownership. Core tests exercise stall exhaustion with short timers. The
SDK's six-minute silence window needs reference-project validation.

## Changelog

Silent chat subscriptions now report `Stream stalled: no records
received` after five stall retries. Network failures retain automatic
recovery, and watch subscriptions remain unlimited.

## Risk

| Condition | Result |
| --- | --- |
| Retryable connection or fetch failure | No finite retry deadline;
exponential backoff continues |
| Repeated connected silence | The sixth 60-second stall ends the
subscription |
| Browser wake or `online` event | Reconnect without consuming or
restoring the stall budget |

With immediate response headers, six silent attempts take 368.5–377
seconds, about 6.1–6.3 minutes. Network delays extend this window. A
healthy tool call with no records can also reach this limit: silence
does not prove that the run is dead. The terminal error stops automatic
client resumption but does not cancel the server-side run.

Shared core consumers retain their configured retry limits. Repeated
successful responses without records now increase backoff until a
decoded record arrives. Caller cancellation and token-refresh limits
remain unchanged.

…, overrides, metrics, and management API

Queues get richer concurrency controls: a total concurrency limit across
all concurrency keys of a queue (alongside the existing per-key limit),
named concurrency limits declared in code and shared across tasks, and
trigger-time selection between them. The dashboard gains a Concurrency
page with per-queue metrics charts and an override dialog for per-key
and total bounds, plus a management API for listing, retrieving,
overriding and resetting concurrency limits. Chat sessions can set
concurrency options when triggering. Run admission gates in the engine
enforce the new limits and are off by default behind
RUN_ENGINE_QUEUE_GATES_ENABLED.

Mono-RevId: b8b2c98fae12cf9abb09b3f1349fd828d053ab70

…run() (#4952)

## Summary

Idempotency keys deduplicate task triggers (`trigger()`,
`triggerAndWait()`, `batchTrigger()`). They do not make the code inside
a task's `run()` idempotent: when a run retries, `run()` executes again
from the top, and any API call inside it happens again unless that call
is itself idempotent.

The idempotency page listed "avoiding double-charging customers" as a
use case, but every example on the page only ever deduplicated a child
trigger, so a reader could reasonably assume a payment call made
directly inside a retryable `run()` was protected. It is not, and no
Trigger.dev-side mechanism can protect it: only the payment provider can
deduplicate its own API call, so the provider's idempotency key has to
be passed too.

This is a docs-only change. No SDK changes.

## What changed

- The intro now states the boundary up front: the key applies to the
trigger call and nothing else.
- The use-case bullets say "trigger the X task once" rather than
implying the side effect itself is deduplicated, and the payments bullet
points at the new section.
- A new "Side effects inside `run()`" section explains why moving the
call into a child task does not close the gap either, and shows a Stripe
refund passing Stripe's `idempotencyKey` alongside a Trigger.dev key for
the follow-up email trigger.

Addresses the documentation side of
[#4627](#4627).
Supersedes
[#4650](#4650), which
was auto-closed by the vouch gate.

Keep steering messages in context across later agent steps and preserve
their original position in saved conversations and subsequent turns.
Persist managed custom response data in stream order and retain prepared
steering input when a conversation resumes in a new worker.

Mono-RevId: 0c642ea561d632dabb68ad024aaff4e7e708d114

Mono-RevId: 5997ba1b23b730bd390acfc3ddd3c8c60f06e00a

)

## Summary

Follow-up to
[#4952](#4952). The new
"Side effects inside `run()`" section told readers to derive the
provider's idempotency key from a payload ID, but the more general
answer is the run ID: `ctx.run.id` is stable across every attempt of a
run, is always available, and is the same value the default `run` scope
already mixes into a Trigger.dev idempotency key.

The Stripe refund example now keys the provider call on `ctx.run.id`,
and a short paragraph explains when a business ID from the payload is
the better choice (when the same task can be triggered more than once
for the same order and you want deduplication across separate runs).

Docs only.

Mono-RevId: b32e17efc094e8641bd7e2ccc544a9c9aeae470f

Add a CI guard that checks new Prisma migrations for idempotent,
lock-safe DDL: creates must use IF NOT EXISTS, drops IF EXISTS, CREATE
TYPE, ADD CONSTRAINT and RENAME must sit in a guarded DO block, INSERTs
need ON CONFLICT, and indexes on existing tables must be built
CONCURRENTLY in a single-statement migration. The enforced cutoff date
is pinned in the `guard:migrations` script in
`apps/webapp/package.json`; `-- --all` audits the whole history locally.
A `-- migration-guard: allow <reason>` comment opts a single statement
out.

Mono-RevId: 7c937700bc4eae8b5b51fecdb60b620bf3c9807b

Mono-RevId: c77f5700f7d4a98e7e2a86685f4a7285960e82cd

…erify tokens

## Summary

Hosted webhook sources can now describe how the provider expects to be
answered, and the ingress honors it. This is the server side for
providers like Discord and WhatsApp, whose SDK sources follow in a later
release.

- A verifier artifact can declare a response contract as data: the
status code for a handshake answer (`respondStatus`), and the codes
returned for accepted deliveries and rejected signatures
(`response.acceptedStatus`, `response.rejectedStatus`). The ingress and
the dashboard's send action map every outcome through the same helper,
so a source that needs 204 on success and 401 on a bad signature gets
exactly that.
- A verifier artifact can declare a GET verification flow
(`getHandshake`) for providers that confirm a callback URL with a
challenge, such as Meta's `hub.challenge`. The ingress answers GET on
the endpoint URL against a dedicated verify token, which you generate
and reveal from the endpoint's Connect panel. The token is its own
credential, separate from the signing secret.
- HMAC verifiers can read the signed timestamp from a body field. The
Linear provider uses it for a 60 second replay window on
`webhookTimestamp`; deliveries are deduplicated on the signed request
itself, so a replay with a different unsigned delivery header is still
recognised as a duplicate.
- The dashboard's test-send re-signs a recorded sample with the current
timestamp, so sources with a body-timestamp replay window accept it.
- An admin API action bootstraps delivery partitions in the configured
webhook database before enablement. It is safe to repeat and preserves
existing partitions. Bootstrap and daily maintenance can use
`WEBHOOK_DATABASE_DIRECT_URL` with separate owner credentials while
application queries use `WEBHOOK_DATABASE_URL`. When the direct URL is
unset, partition operations reuse the webhook writer.
- The index worker protocol gains a message for duplicate webhook ids,
so the CLI can report which files define the same id. The CLI side of
this ships with the SDK release that adds `webhook()`.

Mono-RevId: 54862ef81ec5adc311aa42783020fedc4b8c3ba9

Explain when to collect a tool result with `addToolOutput` and when to
combine `needsApproval` with `execute`. Cover choosing from search
results in the human-in-the-loop guide, including server-side selection
checks and stable operation IDs for retries.

Clarify transcript restoration and the limits of tool-result filtering
for external side effects. Update the frontend approval examples and
link them to the selection guidance.

Mono-RevId: 6c6be1ef71ea4846ba9929179edf48eee77a2af7

Mono-RevId: c893ea0934c69e8fe8ec6f45eaf7be8487e4886b

Clarify chat.agent setup and recovery with examples that check chat
ownership before starting sessions or refreshing tokens, use managed
streamText for steering, and preserve partial responses after failures.
Explain how to load transcripts before resuming the frontend and stop a
resumed generation.

Update the branching guide to use transcript storage and an explicit
active branch, and add checks for transport behavior, error recovery,
and branch isolation.

Mono-RevId: 071c63091f9605d84806381d97311f2fb1583eea

Mono-RevId: b882af2b96296eb4b1010903fe41af0b2b612572

Improve preview branch auto-archive feature with clearer fields,
optional branch exclusions, and smoother modal transitions. Keep branch
creation alongside the search controls while simplifying the empty
state.

Mono-RevId: 8423e6181629877a7971abc9e52cd6fc578a9613

Download saved transcripts from the session inspector when using
built-in storage. Downloads preserve the original stored contents,
including messages and runtime state, and use a `.jsonl` extension for
indexed transcripts.

Mono-RevId: 2bd347196eb5993c27d986ebb7abf460020b670e

…imits

Concurrency limits can now be paused and resumed from the dashboard, the
API, and the SDK (`concurrencyLimits.pause()` /
`concurrencyLimits.resume()`), just like queues. A paused limit stops
every run holding it from being dequeued while keeping its configured
bounds, and resuming restores them.

Mono-RevId: 33cbe4dc731671f083779d05cedac03c3709a654

…rencyLimit

## Summary

Marks `queues.overrideConcurrencyLimit` and
`queues.resetConcurrencyLimit` as deprecated in the SDK. These functions
belong to the legacy model where a queue carried its own concurrency
limit. On the current model a queue is only the ordered line runs wait
in, concurrency is declared on the task with the `concurrency` option,
and limits are managed through `concurrencyLimits.override` and
`concurrencyLimits.reset`. The server already rejects these calls for
queues on the current model.

The `@deprecated` JSDoc includes a migration example:

```ts
export const myTask = task({
  id: "my-task",
  concurrency: { total: 5 },
  run: async (payload) => {
    // ...
  },
});

await concurrencyLimits.override("task/my-task", { total: 10 });
```

Also adds a deprecation callout to the queues docs pointing at the
concurrency docs, plus a changeset.

`queues.pause` and `queues.resume` are intentionally not deprecated:
pausing a queue remains valid flow control on the current model.

Mono-RevId: e75902f159eda3487596ba4d038ed5eedc45bee8

…e dequeue scripts

The run queue's self-heal for saturated concurrency sets is now bounded
and configurable (page size, lock TTL, passes per dequeue, on/off) and
reports its work through OpenTelemetry metrics and span attributes.
Defaults preserve existing behaviour apart from capping passes per
dequeue script at 2.

Mono-RevId: fa05bcd6515f5905f3be52abeebaeeeeb0287617

…ocations atomically

Reallocating purchased concurrency across environments now requires
billing permissions, matching purchases, and allocations are applied
atomically so simultaneous changes can no longer exceed the purchased
amount.

Mono-RevId: 0ff535b7126ef285de9d815196545022ae2236b5

Add `WEBHOOK_DELIVERIES_REPLICATION_DATABASE_URL` so webhook delivery
replication to ClickHouse can use a direct PostgreSQL connection with
its own credentials while application queries use a pooler. When unset,
replication continues to use `WEBHOOK_DATABASE_URL`, falling back to
`DATABASE_URL`.

Mono-RevId: 9991ab7c7dd7df8f00e3a93a94ca2b8ad02bf4cf

The active team members list in organization settings is now sorted
alphabetically instead of appearing in an arbitrary order. Members are
sorted case-insensitively by their display name, falling back to their
email address when no name is set. The pending invites list on the same
page is now sorted alphabetically by email too.

Mono-RevId: 38df496ffef4cf31f8956c4c3cd38387e06e4519