feat(webapp,run-engine,core,clickhouse): concurrency limits management API, metrics and the Concurrency page - #4830
Conversation
|
| Name | Type |
|---|---|
| @trigger.dev/sdk | Patch |
| @trigger.dev/core | Patch |
| @trigger.dev/react-hooks | Patch |
| @trigger.dev/python | Patch |
| @internal/dashboard-agent | Patch |
| @trigger.dev/build | Patch |
| trigger.dev | Patch |
| @trigger.dev/redis-worker | Patch |
| @trigger.dev/schema-to-json | Patch |
| @internal/clickhouse | Patch |
| @internal/llm-model-catalog | Patch |
| @internal/metrics-pipeline | Patch |
| @trigger.dev/rbac | Patch |
| @internal/redis | Patch |
| @internal/replication | Patch |
| @internal/run-engine | Patch |
| @internal/run-store | Patch |
| @internal/schedule-engine | Patch |
| @internal/tracing | Patch |
| @internal/webhook-engine | Patch |
| @internal/webhook-sources | Patch |
| @internal/cache | Patch |
| @trigger.dev/rsc | Patch |
| @trigger.dev/database | Patch |
| @trigger.dev/otlp-importer | Patch |
| @trigger.dev/sso | Patch |
| @internal/testcontainers | Patch |
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe change adds total concurrency fields to queue metrics, ClickHouse aggregates, queue presenters, API responses, and queue views. RunQueue now exposes total concurrency counts and limits, emits total-cap gauge values, and removes per-key limit override handling. Queue and dashboard queries use the Merge Risk: 🟡 Moderate · up to This change adds combined queue concurrency visibility, but the metrics migration can lose aggregate data during view replacement and queue-list pagination can return incomplete pages when ranked entries are stale. These issues should be addressed before merge. 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
8f5ad5d to
c60df90
Compare
@trigger.dev/build
trigger.dev
@trigger.dev/core
@trigger.dev/python
@trigger.dev/react-hooks
@trigger.dev/redis-worker
@trigger.dev/rsc
@trigger.dev/schema-to-json
@trigger.dev/sdk
commit: |
c60df90 to
2816d30
Compare
fe6772f to
6bc8bef
Compare
e802dab to
1627de4
Compare
1627de4 to
f4b8109
Compare
f4b8109 to
ecfbdbb
Compare
Observability mapAs of 19/100 over 459 measured of 478 entry points (base 19, no change) What this PR changed
4 entries removed FIX FIRST
AUDIT 3 of 50 sensitive mutations record an actor. 47 without one. What the score is made ofThe score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md. |
…the concurrency page
…rides converge the engine Limit rows no longer link to the queue detail page: that page is queue observability (queue-scoped metrics, run filters, pause/override actions) and a limit's activity lives on each holder's home queue, so the detail route and retrieve presenter go back to queue rows only. The page's post actions and the detail back-link now target /concurrency directly instead of bouncing through the /queues 301. concurrencyLimits.override now converges the run-engine with Postgres: a failed engine sync compensates from a fresh row read (original error still surfaces), and a successful sync re-checks freshness so a slower older override's engine write can never leave the engine behind a newer one.
… hides on the Concurrency page A limit row's running count spans every concurrency key, so comparing it against the perKey cap tripped the at-limit warning on healthy limits; the warning and health badge now compare against the total bound and stay quiet for perKey-only limits. The paused-environment banner's Manage button hides again on the page it links to, which is /concurrency now.
…lds never read as at-limit The freshness re-check after an override sync converges on the values the engine enforces instead of updatedAt, since Prisma's millisecond updatedAt makes two same-millisecond writes indistinguishable; identical values mean identical engine state so skipping them is safe. A zero at-limit threshold (a paused limit, or a queue overridden to 0) no longer shows the at-limit warning, matching the health badge. The override tests reset mocks per test and assert the exact compensating call so they fail without the fix.
…gument Matching values only prove this actor once synced them; skipping is sound because any actor whose sync diverged the engine also had its compensation fail and its caller received the error.
Failed engine writes surface to the caller, which can retry; a stale write landing after the loop's bounded final read is healed by the next sync or deploy, the same residual the deploy-time queue sync accepts.
…ships Queue reads now report a version that discriminates the shape. V1 queues keep today's fields exactly (their own concurrencyLimit and its override state) so nothing existing breaks; V2 queues are only the line runs wait in and carry no queue-level concurrency, since their limits are read and overridden through concurrencyLimits (concurrencyLimit stays as null so older clients keep parsing). The combined block is gone from the public schema entirely. The public queue routes now serialize through the schema, so dashboard-only extras can never leak onto the wire; the dashboard reads configured bounds from a presenter-attached limits object instead of the public fields.
… as V1 A response without the discriminator has V1 semantics by definition, so the schema defaults it instead of failing the parse against an older server.
…oints reject V2 queues An inline limit on a task's own default queue compiles onto the V2 QUEUE row; its derived task/<id> name now resolves through the concurrency-limits surface (retrieve, override, reset, list) with the queue's own concurrency and length as the running and queued counts, so every declared limit is manageable through one API. The deprecated V1 queue override/reset endpoints reject V2 queues with a pointer to that surface instead of mutating a queue whose response can no longer echo the change; the dashboard's own actions are unaffected. The V2 read arm declares concurrency as never-present so existing reads keep compiling and see undefined, and the pause/override/reset responses serialize through the same schema barrier as the list and retrieve routes.
…d never resolves a stale row Per-key engine writes from the limits surface preserve a queue's pause (the pause is the engine value 0, so an override or reset touching only total no longer silently resumes a paused queue), and a perKey write clears the percent override source so the percent recalculation cannot later clobber or resurrect it. Deploys retire a task's anonymous LIMIT row when its inline limit no longer compiles there, clearing bounds and engine keys so the row cannot shadow the live task/<id> queue row; name resolution and listing skip boundless LIMIT rows accordingly. Boundless V2 queues resolve by name again so an operator can cap an undeclared task through the surface the V1 endpoints point at, while staying out of the list.
…-guarded The retired-row bounds filter applies only to the anonymous limit/task/ namespace: a boundless named limit is a real, deliberately uncapped row (referenced without a declaration) that stays visible and cappable, while a boundless anonymous row is retired and falls through to the live queue row. Each retirement nulls its row guarded on the read updatedAt, so a concurrent operator override or re-creating deploy wins and keeps its own engine sync; engine keys are removed only for rows this deploy actually retired.
…tired rows leave the dashboard Engine cleanup runs first and a failure leaves the row bounded so the next deploy retries, instead of stranding an invisible stale engine limit (worst case a pause-by-zero) after Postgres already cleared. The Concurrency page mirrors the limits surface's namespace scoping: boundless anonymous limit/task rows are retired and hidden while boundless named limits stay visible as real uncapped rows.
…ne state When the guarded null misses because a concurrent override or deploy took the row between the read and the write, the earlier key removal may have erased engine state the winner just synced, including a pause-by-zero; the retire now re-syncs the engine from the fresh row so the winner's bounds stay enforced.
A partial key-removal failure now re-syncs the engine back to the row's bounds (pause-aware) so the engine never half-enforces a limit the row still reports, and both compensating writes settle with any rejection surfaced in the logs instead of being discarded. The row stays bounded on failure, so the next deploy that still names the task retries retirement.
…resh read Both key removals settle before anything else runs, so a sibling removal can never land after the heal and erase it; and the heal re-reads the row instead of trusting the findMany snapshot, so it re-syncs the engine to whatever a concurrent writer persisted rather than stale values.
… not combinedConcurrencyLimit The tooltip still described the pre-redesign cell format and pointed at the removed combinedConcurrencyLimit option; it now mirrors the rendered formats (N per key (M), N total) and names the concurrency option. The queue detail chart hint says total limit instead of combined limit.
… is present With a trailing total hint the cell link is a gap-2 flex container, so the bare number and its label rendered as separate flex items 12px apart. The number and label are one element now, keeping the 4px label gap in every variant.
…ueues that have both On a row with a total bound the dialog shows one field per bound; a blank field leaves that bound unchanged, and removing the override restores both code-declared values. Rows without a total keep the single-limit dialog with the percent toggle.
…ields never override The two-field dialog's remove path treats a never-overridden bound's queue_not_overridden as success, so a total-only or per-key-only override can actually be removed. Fields start blank (current values are placeholders) so blank-means-unchanged holds and a default submit can't pin a bound against future code changes. The total field mirrors the service's environment-limit cap, and a total failure after a per-key write says exactly what applied.
…es bounds strictly The dialog clears both fields whenever it opens so blank-means-unchanged holds beyond the first open. The bounds action parses with Number and an integer check, so scientific-notation input can no longer truncate to a tiny limit, and the remove-failure message only claims the per-key reset happened when it did.
The dialog validates the pair that would be in effect after submit (a blank field keeps its current value) and shows a form error with submit disabled when the per-key limit exceeds the total, since such a per-key limit could never be reached.
…the override dialog Creating a per-key limit above the total is still blocked, but a pair that already conflicts (reachable through the API, which treats the bounds independently) submits improvements with a warning notice instead of trapping the user.
… shrinks it An already-conflicting pair stays editable, but only toward resolution: an edit that widens the gap between the per-key and total limits is blocked like a newly created conflict.
Summary
Stacked on #4829. The Queues page becomes the Concurrency page, and named concurrency limits become fully manageable and observable.
The page keeps its layout, charts and columns at the new
/concurrencyURL (the old/queueslist and detail URLs redirect permanently). Named limits appear as rows alongside queues: running is the limit's live holder count across keyed and keyless runs, queued is an exact counter of runs that must clear the limit to execute, and V2 rows' Limit cell self-labels as "per key" or "total". The self-serve concurrency management page moves to/concurrency-limits.The concurrency limits management API ships here: list, retrieve, override and reset endpoints (
/api/v1/concurrency-limits), with partial overrides (zero pauses a limit), declared values kept as the base, and conflict-safe mutations that survive concurrent deploys, overrides and resets in either direction.Design
The engine's total-concurrency pool now counts keyed and keyless runs together: every admit path (enqueue fast path, base dequeue, queue mover) checks and joins the per-queue group set, and every release path drains it, so a limit's
totalmeans what it says. Each gate queue keeps an exact queued counter (incremented per gate on enqueue, decremented on admit and every queued-removal path, guarded on the zset transition, deduped per run, with an activity- and read-refreshed TTL that re-anchors drift from delta-less windows).Queue metrics gauges (emitted from the dequeue and enqueue Lua scripts) carry total running and the raw stored total limit on base queues as well as keyed ones, flowing through the metrics stream into the ClickHouse rollups for the dashboard's charts.
Also removes the combined-limit override/reset endpoints (queue-level totals are not public API surface) and the response field that exposed them.