Fix rrdcontext metadata leak on non-dbengine hosts - #22438
Conversation
contexts: trigger deep rrdcontext GC from chart-cleanup,
not only dbengine
The deep rrdcontext GC that drops archived RRDINSTANCE
/ RRDMETRIC entries was scheduled only by dbengine
rotation. On RAM-mode (and any non-dbengine) hosts that
schedule never fires, so archived metadata accumulates
with every chart churn -- e.g. k8s cgroup spawn/teardown.
Add rrdcontext_request_full_gc() and invoke it from
svc_rrd_cleanup_obsolete_charts_from_all_hosts when at
least one chart was actually freed. Reuses the existing
120 s debouncer slot, so dbengine hosts are unchanged.
There was a problem hiding this comment.
Pull request overview
This PR aims to prevent unbounded rrdcontext metadata growth on non-dbengine (RAM-mode) hosts by ensuring the “deep” rrdcontext retention recalculation + garbage-collection pass gets scheduled even when dbengine rotations never occur.
Changes:
- Adds a new public API
rrdcontext_request_full_gc()to schedule a deep rrdcontext GC pass using the existing delayed trigger slot. - Updates obsolete-chart cleanup to return a count of freed charts and requests a deep rrdcontext GC when at least one chart was freed.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/database/contexts/rrdcontext.h |
Exposes rrdcontext_request_full_gc() as a public API. |
src/database/contexts/rrdcontext.c |
Implements rrdcontext_request_full_gc() by scheduling the deep retention/GC pass. |
src/daemon/service.c |
Tracks freed charts during cleanup and triggers rrdcontext_request_full_gc() when any charts were freed. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- rrdcontext_request_full_gc() rewrote rrdcontext_next_db_rotation_ut
on every call. Under continuous churn (10 s maintenance loop) the
deadline kept getting pushed out by another 120 s and never
converged, so the deep GC never fired. Only arm when the slot is 0;
the worker zeroes it after running, at which point the next
archival arms a fresh window.
- svc_rrdhost_cleanup_charts_marked_obsolete returned only the
chart-free count. The partial-archive path
(svc_rrdset_archive_obsolete_dimensions -> rrddim_free) archives
dimensions without freeing the surrounding chart and still produces
archived RRDMETRIC entries that need GC. Return
partial_archives + full_archives and trigger on either.
|
@cubic-dev-ai please review again |
@stelfrag I have started the AI code review. It will take a few minutes to complete. |
- svc_rrdset_archive_obsolete_dimensions() now returns the number of
dimensions actually archived this call, not a "fully done" bool.
Callers detect "all candidates archived" by checking
RRDSET_FLAG_OBSOLETE_DIMENSIONS after the call, which is cleared on
entry and only re-set when some candidate could not be archived.
Old contract missed partial-archive cases: archiving 3 of 5 dims
returned false, callers stayed at 0 partial_archives, and the deep
rrdcontext GC trigger never fired even though 3 RRDMETRICs were
already archived.
svc_rrdhost_cleanup_charts_marked_obsolete() now accumulates
archived_items (every dim archived + 1 per chart freed for the
RRDINSTANCE) and returns that as the trigger signal.
- rrdcontext_next_db_rotation_ut is read/written from three threads
(dbengine rotation, service maintenance, rrdcontext worker). All
four access sites now go through __atomic_* with RELAXED ordering,
so 32-bit platforms cannot tear the 64-bit value and the
request_full_gc() check-then-set is a real CAS rather than a
TOCTOU race. RELAXED is sufficient because the slot is a wake hint
and does not publish any data the worker consumes; the worker
re-reads dictionaries under their own locks.
Two paths in rrdcontext_main wrote rrdcontext_next_db_rotation_ut to 0
unconditionally: a pre-clear at the start of
rrdcontext_recalculate_retention_all_hosts() and a final store after
rrdcontext_garbage_collect_for_all_hosts(). With the new
rrdcontext_request_full_gc() CAS'ing 0 -> deadline, a request landing
between the pre-clear and the final store would arm a fresh deadline
that the final unconditional store would silently overwrite, dropping
the request and stranding the just-archived metadata until another
chart-free triggered a new pass.
- Remove the pre-clear in rrdcontext_recalculate_retention_all_hosts:
leave the slot at the deadline value during the pass so concurrent
request_full_gc() calls coalesce into the in-flight pass via their
CAS-fail-on-non-zero semantics.
- Replace the final unconditional store-zero with a CAS expecting
the deadline we observed at the top of the branch. If a concurrent
rrdcontext_db_rotation() (unconditional store) has armed a new
deadline during the pass, leave it in place so the next iteration
fires for whatever was archived after we walked.
The deep-GC pass is now reachable from two trigger paths -- dbengine rotation and chart-cleanup -- but extreme_cardinality.db_rotations was incremented unconditionally inside the worker's combined branch. That made the cardinality-protection guard (`extreme_cardinality.db_rotations && ...`) activate as soon as any chart-cleanup pass ran, even on hosts that never rotated dbengine, which broke its original "wait for first rotation" semantics. Move the increment out of the worker and into rrdcontext_db_rotation() via a small rrdcontext_count_db_rotation() helper. The chart-cleanup trigger (rrdcontext_request_full_gc) deliberately does not call it, so db_rotations now counts only real dbengine rotations, as the guard expects.
There was a problem hiding this comment.
No issues found across 5 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Architecture diagram
sequenceDiagram
participant ChartClean as Chart Cleanup Worker
participant Service as svc_rrd_cleanup_obsolete_charts_from_all_hosts
participant HostClean as svc_rrdhost_cleanup_charts_marked_obsolete
participant DimArchive as svc_rrdset_archive_obsolete_dimensions
participant RRDSet as RRDSET
participant RRDContext as rrdcontext.c
participant GCWorker as RRDContext GC Worker
participant ExtremeCard as extreme_cardinality guard
Note over ChartClean,ExtremeCard: Chart Cleanup → Deep GC Flow
loop Every 10s
ChartClean->>Service: Timer fires
Service->>Service: rrd_rdlock()
loop For each host (non-replicating)
Service->>HostClean: svc_rrdhost_cleanup_charts_marked_obsolete(host)
alt Host has pending obsolete charts/dimensions
HostClean->>HostClean: Iterate RRDSETs
alt Chart has RRDSET_FLAG_OBSOLETE_DIMENSIONS
HostClean->>DimArchive: svc_rrdset_archive_obsolete_dimensions(st, all=false)
DimArchive->>DimArchive: Scan dimensions, archive obsolete
DimArchive->>RRDSet: Archive RRDMETRIC for each dimension
DimArchive-->>HostClean: Return count of archived dimensions
HostClean->>HostClean: archived_items += count
end
alt Chart has RRDSET_FLAG_OBSOLETE (full chart)
HostClean->>DimArchive: svc_rrdset_archive_obsolete_dimensions(st, all=true)
DimArchive->>RRDSet: Archive all dims (archived RRDMETRICs)
DimArchive-->>HostClean: Return count
alt All dimensions archived
HostClean->>HostClean: archived_items += count + 1 (RRDINSTANCE)
HostClean->>RRDSet: rrdset_free(st)
end
end
HostClean-->>Service: Return total archived_items
else No pending work
HostClean-->>Service: Return 0
end
end
Service->>Service: rrd_rdunlock()
alt archived_items > 0
Service->>RRDContext: rrdcontext_request_full_gc()
Note over RRDContext: CAS: only arm if slot == 0
RRDContext->>RRDContext: __atomic_compare_exchange_n(rrdcontext_next_db_rotation_ut, 0, deadline+120s)
alt CAS succeeds (slot was zero)
Note over RRDContext: New GC pass scheduled in 120s
else CAS fails (slot already non-zero)
Note over RRDContext: Existing GC pass covers this request
end
end
end
Note over GCWorker,ExtremeCard: Deep GC Execution (in parallel)
loop Worker loop (every 10s)
GCWorker->>GCWorker: Load rrdcontext_next_db_rotation_ut
alt Deadline reached
GCWorker->>GCWorker: rrdcontext_recalculate_retention_all_hosts()
GCWorker->>GCWorker: rrdcontext_garbage_collect_for_all_hosts()
alt Slot unchanged (no dbengine rotation race)
GCWorker->>GCWorker: CAS slot back to 0
else Slot updated by dbengine rotation
Note over GCWorker: Keep new deadline for next pass
end
end
alt Real dbengine rotation detected
ExtremeCard->>ExtremeCard: extreme_cardinality.db_rotations++
Note over ExtremeCard: Guards extreme cardinality protection
end
end
Note over RRDContext,GCWorker: DBEngine Rotation (unchanged)
RRDContext->>RRDContext: rrdcontext_db_rotation()
RRDContext->>RRDContext: __atomic_store_n(slot, deadline+120s)
RRDContext->>RRDContext: rrdcontext_count_db_rotation() → bumps db_rotations
|
@cubic-dev-ai review this PR |
@stelfrag I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
No issues found across 5 files
Confidence score: 5/5
- Automated review surfaced no issues in the provided summaries.
- No files require special attention.
Architecture diagram
sequenceDiagram
participant ChartClean as Chart Cleanup Worker
participant Service as svc_rrd_cleanup_obsolete_charts_from_all_hosts
participant HostClean as svc_rrdhost_cleanup_charts_marked_obsolete
participant DimArchive as svc_rrdset_archive_obsolete_dimensions
participant RRDSet as RRDSET
participant RRDContext as rrdcontext.c
participant GCWorker as RRDContext GC Worker
participant ExtremeCard as extreme_cardinality guard
Note over ChartClean,ExtremeCard: Chart Cleanup → Deep GC Flow
loop Every 10s
ChartClean->>Service: Timer fires
Service->>Service: rrd_rdlock()
loop For each host (non-replicating)
Service->>HostClean: svc_rrdhost_cleanup_charts_marked_obsolete(host)
alt Host has pending obsolete charts/dimensions
HostClean->>HostClean: Iterate RRDSETs
alt Chart has RRDSET_FLAG_OBSOLETE_DIMENSIONS
HostClean->>DimArchive: svc_rrdset_archive_obsolete_dimensions(st, all=false)
DimArchive->>DimArchive: Scan dimensions, archive obsolete
DimArchive->>RRDSet: Archive RRDMETRIC for each dimension
DimArchive-->>HostClean: Return count of archived dimensions
HostClean->>HostClean: archived_items += count
end
alt Chart has RRDSET_FLAG_OBSOLETE (full chart)
HostClean->>DimArchive: svc_rrdset_archive_obsolete_dimensions(st, all=true)
DimArchive->>RRDSet: Archive all dims (archived RRDMETRICs)
DimArchive-->>HostClean: Return count
alt All dimensions archived
HostClean->>HostClean: archived_items += count + 1 (RRDINSTANCE)
HostClean->>RRDSet: rrdset_free(st)
end
end
HostClean-->>Service: Return total archived_items
else No pending work
HostClean-->>Service: Return 0
end
end
Service->>Service: rrd_rdunlock()
alt archived_items > 0
Service->>RRDContext: rrdcontext_request_full_gc()
Note over RRDContext: CAS: only arm if slot == 0
RRDContext->>RRDContext: __atomic_compare_exchange_n(rrdcontext_next_db_rotation_ut, 0, deadline+120s)
alt CAS succeeds (slot was zero)
Note over RRDContext: New GC pass scheduled in 120s
else CAS fails (slot already non-zero)
Note over RRDContext: Existing GC pass covers this request
end
end
end
Note over GCWorker,ExtremeCard: Deep GC Execution (in parallel)
loop Worker loop (every 10s)
GCWorker->>GCWorker: Load rrdcontext_next_db_rotation_ut
alt Deadline reached
GCWorker->>GCWorker: rrdcontext_recalculate_retention_all_hosts()
GCWorker->>GCWorker: rrdcontext_garbage_collect_for_all_hosts()
alt Slot unchanged (no dbengine rotation race)
GCWorker->>GCWorker: CAS slot back to 0
else Slot updated by dbengine rotation
Note over GCWorker: Keep new deadline for next pass
end
end
alt Real dbengine rotation detected
ExtremeCard->>ExtremeCard: extreme_cardinality.db_rotations++
Note over ExtremeCard: Guards extreme cardinality protection
end
end
Note over RRDContext,GCWorker: DBEngine Rotation (unchanged)
RRDContext->>RRDContext: rrdcontext_db_rotation()
RRDContext->>RRDContext: __atomic_store_n(slot, deadline+120s)
RRDContext->>RRDContext: rrdcontext_count_db_rotation() → bumps db_rotations
…ations
Two follow-ups:
- rrdcontext_request_full_gc() previously dropped requests whose CAS
failed because the worker was already mid-pass with an expired
deadline. The host whose archive triggered the request may have
been walked already, leaving the just-archived metadata stranded
on quiet non-dbengine hosts.
Add a rrdcontext_full_gc_rerun_requested flag, set when the CAS
fails AND the observed deadline is already in the past (mid-pass
case). The worker reads-and-clears the flag at end of pass and
arms a fresh deadline iff the slot is currently 0; concurrent
dbengine_rotation stores survive the follow-up CAS. Future-armed
deadlines don't set the flag because their upcoming pass will see
the archive.
- extreme_cardinality.db_rotations is now atomically incremented
from rrdcontext_count_db_rotation(). The remaining read in
rrdcontext_post_process_updates() was a plain load, mixing atomic
writes with non-atomic reads (a C data race; TSAN would flag it).
Switch to __atomic_load_n with RELAXED ordering.
|
thiagoftsm
left a comment
There was a problem hiding this comment.
PR is running as expected on hoss using db modes dbengine and ram. LGTM!
* Fix rrdcontext metadata leak on non-dbengine hosts
contexts: trigger deep rrdcontext GC from chart-cleanup,
not only dbengine
The deep rrdcontext GC that drops archived RRDINSTANCE
/ RRDMETRIC entries was scheduled only by dbengine
rotation. On RAM-mode (and any non-dbengine) hosts that
schedule never fires, so archived metadata accumulates
with every chart churn -- e.g. k8s cgroup spawn/teardown.
Add rrdcontext_request_full_gc() and invoke it from
svc_rrd_cleanup_obsolete_charts_from_all_hosts when at
least one chart was actually freed. Reuses the existing
120 s debouncer slot, so dbengine hosts are unchanged.
* contexts: keep GC deadline armed and trigger on any archival
- rrdcontext_request_full_gc() rewrote rrdcontext_next_db_rotation_ut
on every call. Under continuous churn (10 s maintenance loop) the
deadline kept getting pushed out by another 120 s and never
converged, so the deep GC never fired. Only arm when the slot is 0;
the worker zeroes it after running, at which point the next
archival arms a fresh window.
- svc_rrdhost_cleanup_charts_marked_obsolete returned only the
chart-free count. The partial-archive path
(svc_rrdset_archive_obsolete_dimensions -> rrddim_free) archives
dimensions without freeing the surrounding chart and still produces
archived RRDMETRIC entries that need GC. Return
partial_archives + full_archives and trigger on either.
* contexts: count actual archives and atomicize the GC schedule slot
- svc_rrdset_archive_obsolete_dimensions() now returns the number of
dimensions actually archived this call, not a "fully done" bool.
Callers detect "all candidates archived" by checking
RRDSET_FLAG_OBSOLETE_DIMENSIONS after the call, which is cleared on
entry and only re-set when some candidate could not be archived.
Old contract missed partial-archive cases: archiving 3 of 5 dims
returned false, callers stayed at 0 partial_archives, and the deep
rrdcontext GC trigger never fired even though 3 RRDMETRICs were
already archived.
svc_rrdhost_cleanup_charts_marked_obsolete() now accumulates
archived_items (every dim archived + 1 per chart freed for the
RRDINSTANCE) and returns that as the trigger signal.
- rrdcontext_next_db_rotation_ut is read/written from three threads
(dbengine rotation, service maintenance, rrdcontext worker). All
four access sites now go through __atomic_* with RELAXED ordering,
so 32-bit platforms cannot tear the 64-bit value and the
request_full_gc() check-then-set is a real CAS rather than a
TOCTOU race. RELAXED is sufficient because the slot is a wake hint
and does not publish any data the worker consumes; the worker
re-reads dictionaries under their own locks.
* contexts: don't drop GC requests racing the worker's clear pass
Two paths in rrdcontext_main wrote rrdcontext_next_db_rotation_ut to 0
unconditionally: a pre-clear at the start of
rrdcontext_recalculate_retention_all_hosts() and a final store after
rrdcontext_garbage_collect_for_all_hosts(). With the new
rrdcontext_request_full_gc() CAS'ing 0 -> deadline, a request landing
between the pre-clear and the final store would arm a fresh deadline
that the final unconditional store would silently overwrite, dropping
the request and stranding the just-archived metadata until another
chart-free triggered a new pass.
- Remove the pre-clear in rrdcontext_recalculate_retention_all_hosts:
leave the slot at the deadline value during the pass so concurrent
request_full_gc() calls coalesce into the in-flight pass via their
CAS-fail-on-non-zero semantics.
- Replace the final unconditional store-zero with a CAS expecting
the deadline we observed at the top of the branch. If a concurrent
rrdcontext_db_rotation() (unconditional store) has armed a new
deadline during the pass, leave it in place so the next iteration
fires for whatever was archived after we walked.
* Address review comments
* Address review comments
* contexts: keep extreme-cardinality guard on real dbengine rotations only
The deep-GC pass is now reachable from two trigger paths -- dbengine
rotation and chart-cleanup -- but extreme_cardinality.db_rotations was
incremented unconditionally inside the worker's combined branch. That
made the cardinality-protection guard
(`extreme_cardinality.db_rotations && ...`) activate as soon as any
chart-cleanup pass ran, even on hosts that never rotated dbengine,
which broke its original "wait for first rotation" semantics.
Move the increment out of the worker and into rrdcontext_db_rotation()
via a small rrdcontext_count_db_rotation() helper. The chart-cleanup
trigger (rrdcontext_request_full_gc) deliberately does not call it, so
db_rotations now counts only real dbengine rotations, as the guard
expects.
* contexts: don't drop GC requests landing mid-pass; atomic-load db_rotations
Two follow-ups:
- rrdcontext_request_full_gc() previously dropped requests whose CAS
failed because the worker was already mid-pass with an expired
deadline. The host whose archive triggered the request may have
been walked already, leaving the just-archived metadata stranded
on quiet non-dbengine hosts.
Add a rrdcontext_full_gc_rerun_requested flag, set when the CAS
fails AND the observed deadline is already in the past (mid-pass
case). The worker reads-and-clears the flag at end of pass and
arms a fresh deadline iff the slot is currently 0; concurrent
dbengine_rotation stores survive the follow-up CAS. Future-armed
deadlines don't set the flag because their upcoming pass will see
the archive.
- extreme_cardinality.db_rotations is now atomically incremented
from rrdcontext_count_db_rotation(). The remaining read in
rrdcontext_post_process_updates() was a plain load, mixing atomic
writes with non-atomic reads (a C data race; TSAN would flag it).
Switch to __atomic_load_n with RELAXED ordering.
(cherry picked from commit 11c8088)



Summary
On RAM-mode agents (eg k8s iot children), the deep rrdcontext GC that drops archived
RRDINSTANCE/RRDMETRICentries was only scheduled by dbengine on database rotation.With no dbengine, that schedule never fires, so context metadata piles up forever as charts churn
This PR adds
rrdcontext_request_full_gc()and calls it from the chart-cleanup pass whenever at least one chart was actually freed. It reuses the existing 120 s debouncer slot so concurrent triggers coalesce and dbengine hosts behave identically.Summary by cubic
Fixes a rrdcontext metadata leak on non-dbengine hosts by scheduling deep GC whenever chart cleanup archives charts or dimensions. Prevents unbounded growth of archived
RRDINSTANCE/RRDMETRICon RAM‑mode hosts (e.g., k8s).rrdcontext_request_full_gc(); reuses the 120 s debounce and arms only when the slot is 0, so churn can’t push it out forever.db_rotationsupdated and read atomically.Written for commit e811cf6. Summary will update on new commits.