Skip to content

feat(webapp,run-engine,core,clickhouse): concurrency limits management API, metrics and the Concurrency page - #4830

Open
matt-aitken wants to merge 86 commits into
feat/queue-concurrency-overridesfrom
feat/queue-concurrency-metrics
Open

matt-aitken wants to merge 86 commits into
feat/queue-concurrency-overridesfrom
feat/queue-concurrency-metrics

Conversation

@matt-aitken

@matt-aitken matt-aitken commented Aug 29, 2026

Copy link
Copy Markdown
Member

Summary

Stacked on #4829. The Queues page becomes the Concurrency page, and named concurrency limits become fully manageable and observable.

The page keeps its layout, charts and columns at the new /concurrency URL (the old /queues list and detail URLs redirect permanently). Named limits appear as rows alongside queues: running is the limit's live holder count across keyed and keyless runs, queued is an exact counter of runs that must clear the limit to execute, and V2 rows' Limit cell self-labels as "per key" or "total". The self-serve concurrency management page moves to /concurrency-limits.

The concurrency limits management API ships here: list, retrieve, override and reset endpoints (/api/v1/concurrency-limits), with partial overrides (zero pauses a limit), declared values kept as the base, and conflict-safe mutations that survive concurrent deploys, overrides and resets in either direction.

Design

CleanShot 2026-09-17 at 17 35 49@2x

The engine's total-concurrency pool now counts keyed and keyless runs together: every admit path (enqueue fast path, base dequeue, queue mover) checks and joins the per-queue group set, and every release path drains it, so a limit's total means what it says. Each gate queue keeps an exact queued counter (incremented per gate on enqueue, decremented on admit and every queued-removal path, guarded on the zset transition, deduped per run, with an activity- and read-refreshed TTL that re-anchors drift from delta-less windows).

Queue metrics gauges (emitted from the dequeue and enqueue Lua scripts) carry total running and the raw stored total limit on base queues as well as keyed ones, flowing through the metrics stream into the ClickHouse rollups for the dashboard's charts.

Also removes the combined-limit override/reset endpoints (queue-level totals are not public API surface) and the response field that exposed them.

@changeset-bot

changeset-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 3a11b20

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes changesets to release 27 packages
Name Type
@trigger.dev/sdk Patch
@trigger.dev/core Patch
@trigger.dev/react-hooks Patch
@trigger.dev/python Patch
@internal/dashboard-agent Patch
@trigger.dev/build Patch
trigger.dev Patch
@trigger.dev/redis-worker Patch
@trigger.dev/schema-to-json Patch
@internal/clickhouse Patch
@internal/llm-model-catalog Patch
@internal/metrics-pipeline Patch
@trigger.dev/rbac Patch
@internal/redis Patch
@internal/replication Patch
@internal/run-engine Patch
@internal/run-store Patch
@internal/schedule-engine Patch
@internal/tracing Patch
@internal/webhook-engine Patch
@internal/webhook-sources Patch
@internal/cache Patch
@trigger.dev/rsc Patch
@trigger.dev/database Patch
@trigger.dev/otlp-importer Patch
@trigger.dev/sso Patch
@internal/testcontainers Patch

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The change adds total concurrency fields to queue metrics, ClickHouse aggregates, queue presenters, API responses, and queue views. RunQueue now exposes total concurrency counts and limits, emits total-cap gauge values, and removes per-key limit override handling. Queue and dashboard queries use the concurrency_metrics tables. Queue views display combined limits and a combined concurrency chart. Metrics tests now wait for emitter readiness and validate gauge values.

Merge Risk: 🟡 Moderate · up to a20ab

This change adds combined queue concurrency visibility, but the metrics migration can lose aggregate data during view replacement and queue-list pagination can return incomplete pages when ranked entries are stale. These issues should be addressed before merge.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 58.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 24 functions across 28 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check ⚠️ Warning The description gives a detailed and relevant change summary, but it omits the required issue closure, checklist, Testing, Changelog, and Screenshots headings from the template. Add the required template sections. Include a valid "Closes #" reference, complete the checklist, document testing steps, add a short Changelog entry, and include a Screenshots section or state that screenshots are not applicable.
✅ Passed checks (3 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main changes: concurrency limits management, metrics, and the Concurrency page.
  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/queue-concurrency-metrics

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from 8f5ad5d to c60df90 Compare August 29, 2026 14:49
@pkg-pr-new

pkg-pr-new Bot commented Aug 29, 2026

Copy link
Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@e7eb809

trigger.dev

npm i https://pkg.pr.new/trigger.dev@e7eb809

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@e7eb809

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@e7eb809

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@e7eb809

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@e7eb809

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@e7eb809

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@e7eb809

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@e7eb809

commit: e7eb809

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from c60df90 to 2816d30 Compare August 29, 2026 15:33

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from fe6772f to 6bc8bef Compare August 29, 2026 15:52

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from e802dab to 1627de4 Compare August 29, 2026 18:11
@matt-aitken matt-aitken changed the title feat(webapp,run-engine,core,clickhouse): surface total concurrency in metrics and dashboard feat(webapp,run-engine,core,clickhouse): surface combined concurrency in metrics and dashboard Aug 29, 2026
devin-ai-integration[bot]

This comment was marked as resolved.

@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from 1627de4 to f4b8109 Compare August 29, 2026 18:18
@matt-aitken
matt-aitken force-pushed the feat/queue-concurrency-metrics branch from f4b8109 to ecfbdbb Compare August 29, 2026 18:21
devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

@github-actions

github-actions Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Observability map

As of 6ba30c7.

19/100 over 459 measured of 478 entry points (base 19, no change)

What this PR changed

route base head now failing
/app/orgs/:organizationSlug/projects/:projectParam/env/:envParam/concurrency/:queueParam new 0 request-context
/_app/orgs/:organizationSlug/projects/:projectParam/env/:envParam/concurrency-limits new 0 request-context
/_app/orgs/:organizationSlug/projects/:projectParam/env/:envParam/queues/: new 0 request-context
/api/v1/concurrency-limits new 0 request-context
/api/v1/concurrency-limits/:name new 0 request-context
/api/v1/concurrency-limits/:name/override new 0 request-context
/api/v1/concurrency-limits/:name/reset new 0 request-context
/_app/orgs/:organizationSlug/projects/:projectParam/env/:envParam/queues new not measured

4 entries removed

FIX FIRST

  • /api/v1/projects/:projectRef/envvars (sensitive) - auth-boundary, request-context
  • /auth/sso (sensitive) - auth-boundary, request-context
  • /_app/orgs/:organizationSlug/settings/team (sensitive) - error-classification, auth-scope, request-context

AUDIT 3 of 50 sensitive mutations record an actor. 47 without one.
CONTEXT 23 of 459 entry points name a tenant on a failure path. 356 appear only here, 39 of them sensitive, in the JSON rather than the fix list.

What the score is made of
CHECKS
  error-classification  185 applicable, 106 pass,   0 sole, global without it 11
  auth-boundary          62 applicable,  57 pass,   0 sole, global without it 16
  auth-scope             19 applicable,  17 pass,   0 sole, global without it 19
  request-context       459 applicable,  23 pass, 252 sole, global without it 64
  audit-trail            50 applicable,   3 pass,   0 sole, not in the score

The score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md.

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 0 new potential issues.

Devin Review

…rides converge the engine

Limit rows no longer link to the queue detail page: that page is queue
observability (queue-scoped metrics, run filters, pause/override actions)
and a limit's activity lives on each holder's home queue, so the detail
route and retrieve presenter go back to queue rows only. The page's post
actions and the detail back-link now target /concurrency directly instead
of bouncing through the /queues 301.

concurrencyLimits.override now converges the run-engine with Postgres: a
failed engine sync compensates from a fresh row read (original error still
surfaces), and a successful sync re-checks freshness so a slower older
override's engine write can never leave the engine behind a newer one.
… hides on the Concurrency page

A limit row's running count spans every concurrency key, so comparing it
against the perKey cap tripped the at-limit warning on healthy limits;
the warning and health badge now compare against the total bound and stay
quiet for perKey-only limits. The paused-environment banner's Manage
button hides again on the page it links to, which is /concurrency now.
…lds never read as at-limit

The freshness re-check after an override sync converges on the values the
engine enforces instead of updatedAt, since Prisma's millisecond updatedAt
makes two same-millisecond writes indistinguishable; identical values mean
identical engine state so skipping them is safe. A zero at-limit threshold
(a paused limit, or a queue overridden to 0) no longer shows the at-limit
warning, matching the health badge. The override tests reset mocks per test
and assert the exact compensating call so they fail without the fix.
…gument

Matching values only prove this actor once synced them; skipping is sound
because any actor whose sync diverged the engine also had its compensation
fail and its caller received the error.
Failed engine writes surface to the caller, which can retry; a stale write
landing after the loop's bounded final read is healed by the next sync or
deploy, the same residual the deploy-time queue sync accepts.
…ships

Queue reads now report a version that discriminates the shape. V1 queues
keep today's fields exactly (their own concurrencyLimit and its override
state) so nothing existing breaks; V2 queues are only the line runs wait
in and carry no queue-level concurrency, since their limits are read and
overridden through concurrencyLimits (concurrencyLimit stays as null so
older clients keep parsing). The combined block is gone from the public
schema entirely. The public queue routes now serialize through the schema,
so dashboard-only extras can never leak onto the wire; the dashboard reads
configured bounds from a presenter-attached limits object instead of the
public fields.
… as V1

A response without the discriminator has V1 semantics by definition, so the
schema defaults it instead of failing the parse against an older server.
…oints reject V2 queues

An inline limit on a task's own default queue compiles onto the V2 QUEUE
row; its derived task/<id> name now resolves through the concurrency-limits
surface (retrieve, override, reset, list) with the queue's own concurrency
and length as the running and queued counts, so every declared limit is
manageable through one API. The deprecated V1 queue override/reset
endpoints reject V2 queues with a pointer to that surface instead of
mutating a queue whose response can no longer echo the change; the
dashboard's own actions are unaffected. The V2 read arm declares
concurrency as never-present so existing reads keep compiling and see
undefined, and the pause/override/reset responses serialize through the
same schema barrier as the list and retrieve routes.
…d never resolves a stale row

Per-key engine writes from the limits surface preserve a queue's pause (the
pause is the engine value 0, so an override or reset touching only total no
longer silently resumes a paused queue), and a perKey write clears the
percent override source so the percent recalculation cannot later clobber
or resurrect it. Deploys retire a task's anonymous LIMIT row when its
inline limit no longer compiles there, clearing bounds and engine keys so
the row cannot shadow the live task/<id> queue row; name resolution and
listing skip boundless LIMIT rows accordingly. Boundless V2 queues resolve
by name again so an operator can cap an undeclared task through the surface
the V1 endpoints point at, while staying out of the list.
…-guarded

The retired-row bounds filter applies only to the anonymous limit/task/
namespace: a boundless named limit is a real, deliberately uncapped row
(referenced without a declaration) that stays visible and cappable, while a
boundless anonymous row is retired and falls through to the live queue row.
Each retirement nulls its row guarded on the read updatedAt, so a concurrent
operator override or re-creating deploy wins and keeps its own engine sync;
engine keys are removed only for rows this deploy actually retired.
…tired rows leave the dashboard

Engine cleanup runs first and a failure leaves the row bounded so the next
deploy retries, instead of stranding an invisible stale engine limit (worst
case a pause-by-zero) after Postgres already cleared. The Concurrency page
mirrors the limits surface's namespace scoping: boundless anonymous
limit/task rows are retired and hidden while boundless named limits stay
visible as real uncapped rows.
…ne state

When the guarded null misses because a concurrent override or deploy took
the row between the read and the write, the earlier key removal may have
erased engine state the winner just synced, including a pause-by-zero; the
retire now re-syncs the engine from the fresh row so the winner's bounds
stay enforced.
A partial key-removal failure now re-syncs the engine back to the row's
bounds (pause-aware) so the engine never half-enforces a limit the row
still reports, and both compensating writes settle with any rejection
surfaced in the logs instead of being discarded. The row stays bounded on
failure, so the next deploy that still names the task retries retirement.
…resh read

Both key removals settle before anything else runs, so a sibling removal
can never land after the heal and erase it; and the heal re-reads the row
instead of trusting the findMany snapshot, so it re-syncs the engine to
whatever a concurrent writer persisted rather than stale values.
… not combinedConcurrencyLimit

The tooltip still described the pre-redesign cell format and pointed at the
removed combinedConcurrencyLimit option; it now mirrors the rendered formats
(N per key (M), N total) and names the concurrency option. The queue detail
chart hint says total limit instead of combined limit.
… is present

With a trailing total hint the cell link is a gap-2 flex container, so the
bare number and its label rendered as separate flex items 12px apart. The
number and label are one element now, keeping the 4px label gap in every
variant.
…ueues that have both

On a row with a total bound the dialog shows one field per bound; a blank
field leaves that bound unchanged, and removing the override restores both
code-declared values. Rows without a total keep the single-limit dialog with
the percent toggle.
devin-ai-integration[bot]

This comment was marked as resolved.

…ields never override

The two-field dialog's remove path treats a never-overridden bound's
queue_not_overridden as success, so a total-only or per-key-only override
can actually be removed. Fields start blank (current values are placeholders)
so blank-means-unchanged holds and a default submit can't pin a bound against
future code changes. The total field mirrors the service's environment-limit
cap, and a total failure after a per-key write says exactly what applied.
…es bounds strictly

The dialog clears both fields whenever it opens so blank-means-unchanged
holds beyond the first open. The bounds action parses with Number and an
integer check, so scientific-notation input can no longer truncate to a tiny
limit, and the remove-failure message only claims the per-key reset happened
when it did.
The dialog validates the pair that would be in effect after submit (a blank
field keeps its current value) and shows a form error with submit disabled
when the per-key limit exceeds the total, since such a per-key limit could
never be reached.
devin-ai-integration[bot]

This comment was marked as resolved.

…the override dialog

Creating a per-key limit above the total is still blocked, but a pair that
already conflicts (reachable through the API, which treats the bounds
independently) submits improvements with a warning notice instead of trapping
the user.
devin-ai-integration[bot]

This comment was marked as resolved.

… shrinks it

An already-conflicting pair stays editable, but only toward resolution: an
edit that widens the gap between the per-key and total limits is blocked like
a newly created conflict.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants