Skip to content

perf: store workspace agent session counts as JSONB - #28126

Merged
EhabY merged 1 commit into
mainfrom
perf/workspace-agent-session-counts-jsonb
Sep 7, 2026
Merged

EhabY merged 1 commit into
mainfrom
perf/workspace-agent-session-counts-jsonb

Conversation

@EhabY

@EhabY EhabY commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Replaces the four session_count_* columns on workspace_agent_stats with a sparse session_counts JSONB object, so counting a new app needs no schema change.

  • Only positive counts are stored, so an idle agent writes {} instead of four zeroes, and the template insights covering index drops four bigints from its INCLUDE list.
  • The batcher normalizes and caps outside its lock, still ingests the deprecated fields from 2.10 agents, and marshals one object per row.
  • Every reader and rollup moves to JSONB. Output column names and API shapes are unchanged.
  • Resolves the session_count_sftp TODOs: a new session type is now a new key.

Migration

Converts the window DeleteOldWorkspaceAgentStats retains, MAX(template_usage_stats.start_time) minus one day. Anything older is already rolled up. When nothing has ever rolled up there is no watermark to fall back on, so every row converts instead of letting old activity roll up as zero minutes later.

A backlog the rollup never consumed spanning more than 24 hours usually means the rollup is failing or blocked. The migration proceeds anyway and emits a warning naming the span, with a hint to check coderd logs and pg_locks after the upgrade:

WARNING: migration 000590 found 30 days of workspace agent stats that template usage stats never rolled up

Measured on 10M rows: a healthy rollup converts ~330k rows in 11s on PG13 in Docker (4.44s on PG17 with tmpfs, 6.13s on durable storage). A rollup 30 days behind, or a deployment that never rolled up, converts all 10M rows in about 3.5 minutes, roughly 21 seconds per million backlogged rows.

Trade-offs

  • The conversion is unbounded and holds ACCESS EXCLUSIVE, so agent stat writes block for its duration; DROP COLUMN and the index rebuild need the lock regardless. A deployment whose rollup has failed for months converts its whole backlog at ~21s per million rows. The alternative, aborting until the rollup is fixed, blocked the upgrade with no way to trigger a rollup on demand.
  • jsonb parsing on a hot read path. UpsertTemplateUsageStats extracts keys per row, every five minutes over the retained window, where it compared four integers. Neither form was indexed; the cost buys schema-free new apps.
  • Three queries change plan shape, not just their column expressions, so a regression is harder to attribute. Family grouping in feat: attribute session counts by app family #28337 needs one pass per row, and doing it here avoids rewriting the same blocks twice.
  • One PR. The migration drops the columns, so splitting it leaves either old code on the new schema or new queries on a missing column.
  • Rollback discards unknown keys. The down migration restores the four known fields only; nothing else has a column to land in.

Depends on #28125. Phase 3 of #27410.

@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from b654c59 to 102d860 Compare August 13, 2026 16:29
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from 102d860 to ec55b67 Compare August 18, 2026 13:05
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from ec55b67 to a105341 Compare August 19, 2026 11:19
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from 9009c59 to 5b4c776 Compare August 19, 2026 14:14
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from 5b4c776 to 88ce165 Compare August 19, 2026 23:25
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from 88ce165 to ae67674 Compare August 19, 2026 23:43
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from ae67674 to d705373 Compare August 20, 2026 12:29
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from d705373 to 04e3ac5 Compare August 20, 2026 23:22
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from 04e3ac5 to 0ed88b8 Compare August 20, 2026 23:41
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from 0ed88b8 to f163971 Compare August 20, 2026 23:54
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from f163971 to a20e937 Compare August 21, 2026 00:04
Base automatically changed from feat/agent-session-counts-protocol to main August 21, 2026 00:14
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch 3 times, most recently from 99052ce to 93c505d Compare August 23, 2026 14:26
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch 4 times, most recently from 8caca66 to 32f9211 Compare August 23, 2026 18:00
@EhabY
EhabY marked this pull request as ready for review August 23, 2026 19:06
@EhabY
EhabY requested a review from code-asher August 23, 2026 19:06
@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch 2 times, most recently from 3b8fc8d to 5b095c0 Compare August 25, 2026 15:24

@code-asher code-asher left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only got through a couple files, will finish review tomorrow!

Comment thread coderd/database/dbgen/dbgen.go Outdated
@EhabY
EhabY requested a review from code-asher August 28, 2026 13:08

@code-asher code-asher left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! I messed around with connecting clients as well and everything seems to be reported properly.

@code-asher

code-asher commented Aug 31, 2026

Copy link
Copy Markdown
Member

Ah actually I was going to comment on the warning. Do you know how/if that would surface to the end user? Outputs to stderr or something? I briefly tried testing with the develop script but possibly it swallows some output.

@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from 3941985 to bcb1c60 Compare September 1, 2026 23:22
Replaces the four session_count_* columns on workspace_agent_stats with
a sparse session_counts JSONB object, so counting a new app needs no
schema change. Only positive counts are stored, every reader and rollup
moves to JSONB, and output column names and API shapes are unchanged.

The migration converts every row the rollup has not consumed, however
far behind it is, and warns when that backlog spans more than 24 hours.
Without a rollup watermark every row converts.
@EhabY

EhabY commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

This follows the existing migration warning behavior, including migration 000563. It can appear in PostgreSQL's logs, but we don't currently forward it into coderd’s normal warning output.

Since it’s advisory and the migration proceeds without intervention, I'd leave that behavior unchanged here and handle notice forwarding separately if we want it.

@EhabY
EhabY force-pushed the perf/workspace-agent-session-counts-jsonb branch from bcb1c60 to c8d5de0 Compare September 7, 2026 20:11
@EhabY
EhabY merged commit 90c7533 into main Sep 7, 2026
24 checks passed
@EhabY
EhabY deleted the perf/workspace-agent-session-counts-jsonb branch September 7, 2026 20:31
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 7, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants