Skip to content

Fix rrdcontext metadata leak on non-dbengine hosts - #22438

Merged
stelfrag merged 8 commits into
netdata:masterfrom
stelfrag:rrdcontext-non-dbengine-gc
May 8, 2026
Merged

Fix rrdcontext metadata leak on non-dbengine hosts#22438
stelfrag merged 8 commits into
netdata:masterfrom
stelfrag:rrdcontext-non-dbengine-gc

Conversation

@stelfrag

@stelfrag stelfrag commented May 7, 2026

Copy link
Copy Markdown
Collaborator
Summary

On RAM-mode agents (eg k8s iot children), the deep rrdcontext GC that drops archived RRDINSTANCE / RRDMETRIC entries was only scheduled by dbengine on database rotation.

With no dbengine, that schedule never fires, so context metadata piles up forever as charts churn

This PR adds rrdcontext_request_full_gc() and calls it from the chart-cleanup pass whenever at least one chart was actually freed. It reuses the existing 120 s debouncer slot so concurrent triggers coalesce and dbengine hosts behave identically.


Summary by cubic

Fixes a rrdcontext metadata leak on non-dbengine hosts by scheduling deep GC whenever chart cleanup archives charts or dimensions. Prevents unbounded growth of archived RRDINSTANCE/RRDMETRIC on RAM‑mode hosts (e.g., k8s).

  • Bug Fixes
    • Added rrdcontext_request_full_gc(); reuses the 120 s debounce and arms only when the slot is 0, so churn can’t push it out forever.
    • Chart cleanup now counts actual archives (each dimension archived + 1 per chart freed) across hosts and requests GC when >0, including partial‑dimension archives on live charts.
    • Hardened the GC schedule slot: atomic load/store/CAS, no pre‑clear during a pass, and the worker clears via CAS only if the same deadline; mid‑pass requests set a rerun flag so a follow‑up is armed; the extreme‑cardinality guard now activates only on real dbengine rotations with db_rotations updated and read atomically.

Written for commit e811cf6. Summary will update on new commits.

  contexts: trigger deep rrdcontext GC from chart-cleanup,
  not only dbengine

    The deep rrdcontext GC that drops archived RRDINSTANCE
    / RRDMETRIC entries was scheduled only by dbengine
    rotation. On RAM-mode (and any non-dbengine) hosts that
    schedule never fires, so archived metadata accumulates
    with every chart churn -- e.g. k8s cgroup spawn/teardown.

    Add rrdcontext_request_full_gc() and invoke it from
    svc_rrd_cleanup_obsolete_charts_from_all_hosts when at
    least one chart was actually freed. Reuses the existing
    120 s debouncer slot, so dbengine hosts are unchanged.
@stelfrag
stelfrag marked this pull request as ready for review May 7, 2026 09:50
@stelfrag
stelfrag requested a review from thiagoftsm as a code owner May 7, 2026 09:50
Copilot AI review requested due to automatic review settings May 7, 2026 09:50
@stelfrag
stelfrag requested a review from vkalintiris as a code owner May 7, 2026 09:50
@stelfrag
stelfrag marked this pull request as draft May 7, 2026 09:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to prevent unbounded rrdcontext metadata growth on non-dbengine (RAM-mode) hosts by ensuring the “deep” rrdcontext retention recalculation + garbage-collection pass gets scheduled even when dbengine rotations never occur.

Changes:

  • Adds a new public API rrdcontext_request_full_gc() to schedule a deep rrdcontext GC pass using the existing delayed trigger slot.
  • Updates obsolete-chart cleanup to return a count of freed charts and requests a deep rrdcontext GC when at least one chart was freed.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
src/database/contexts/rrdcontext.h Exposes rrdcontext_request_full_gc() as a public API.
src/database/contexts/rrdcontext.c Implements rrdcontext_request_full_gc() by scheduling the deep retention/GC pass.
src/daemon/service.c Tracks freed charts during cleanup and triggers rrdcontext_request_full_gc() when any charts were freed.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/database/contexts/rrdcontext.c Outdated
Comment thread src/daemon/service.c Outdated
    - rrdcontext_request_full_gc() rewrote rrdcontext_next_db_rotation_ut
      on every call. Under continuous churn (10 s maintenance loop) the
      deadline kept getting pushed out by another 120 s and never
      converged, so the deep GC never fired. Only arm when the slot is 0;
      the worker zeroes it after running, at which point the next
      archival arms a fresh window.

    - svc_rrdhost_cleanup_charts_marked_obsolete returned only the
      chart-free count. The partial-archive path
      (svc_rrdset_archive_obsolete_dimensions -> rrddim_free) archives
      dimensions without freeing the surrounding chart and still produces
      archived RRDMETRIC entries that need GC. Return
      partial_archives + full_archives and trigger on either.
@stelfrag

stelfrag commented May 7, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai please review again

@cubic-dev-ai

cubic-dev-ai Bot commented May 7, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai please review again

@stelfrag I have started the AI code review. It will take a few minutes to complete.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comment thread src/daemon/service.c Outdated
Comment thread src/database/contexts/rrdcontext.c Outdated
    - svc_rrdset_archive_obsolete_dimensions() now returns the number of
      dimensions actually archived this call, not a "fully done" bool.
      Callers detect "all candidates archived" by checking
      RRDSET_FLAG_OBSOLETE_DIMENSIONS after the call, which is cleared on
      entry and only re-set when some candidate could not be archived.

      Old contract missed partial-archive cases: archiving 3 of 5 dims
      returned false, callers stayed at 0 partial_archives, and the deep
      rrdcontext GC trigger never fired even though 3 RRDMETRICs were
      already archived.

      svc_rrdhost_cleanup_charts_marked_obsolete() now accumulates
      archived_items (every dim archived + 1 per chart freed for the
      RRDINSTANCE) and returns that as the trigger signal.

    - rrdcontext_next_db_rotation_ut is read/written from three threads
      (dbengine rotation, service maintenance, rrdcontext worker). All
      four access sites now go through __atomic_* with RELAXED ordering,
      so 32-bit platforms cannot tear the 64-bit value and the
      request_full_gc() check-then-set is a real CAS rather than a
      TOCTOU race. RELAXED is sufficient because the slot is a wake hint
      and does not publish any data the worker consumes; the worker
      re-reads dictionaries under their own locks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread src/database/contexts/rrdcontext-worker.c Outdated
  Two paths in rrdcontext_main wrote rrdcontext_next_db_rotation_ut to 0
  unconditionally: a pre-clear at the start of
  rrdcontext_recalculate_retention_all_hosts() and a final store after
  rrdcontext_garbage_collect_for_all_hosts(). With the new
  rrdcontext_request_full_gc() CAS'ing 0 -> deadline, a request landing
  between the pre-clear and the final store would arm a fresh deadline
  that the final unconditional store would silently overwrite, dropping
  the request and stranding the just-archived metadata until another
  chart-free triggered a new pass.

    - Remove the pre-clear in rrdcontext_recalculate_retention_all_hosts:
      leave the slot at the deadline value during the pass so concurrent
      request_full_gc() calls coalesce into the in-flight pass via their
      CAS-fail-on-non-zero semantics.

    - Replace the final unconditional store-zero with a CAS expecting
      the deadline we observed at the top of the branch. If a concurrent
      rrdcontext_db_rotation() (unconditional store) has armed a new
      deadline during the pass, leave it in place so the next iteration
      fires for whatever was archived after we walked.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@stelfrag
stelfrag requested a review from Copilot May 7, 2026 13:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

Comment thread src/database/contexts/rrdcontext-worker.c Outdated
Comment thread src/daemon/service.c Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

Comment thread src/database/contexts/rrdcontext-worker.c
  The deep-GC pass is now reachable from two trigger paths -- dbengine
  rotation and chart-cleanup -- but extreme_cardinality.db_rotations was
  incremented unconditionally inside the worker's combined branch. That
  made the cardinality-protection guard
  (`extreme_cardinality.db_rotations && ...`) activate as soon as any
  chart-cleanup pass ran, even on hosts that never rotated dbengine,
  which broke its original "wait for first rotation" semantics.

  Move the increment out of the worker and into rrdcontext_db_rotation()
  via a small rrdcontext_count_db_rotation() helper. The chart-cleanup
  trigger (rrdcontext_request_full_gc) deliberately does not call it, so
  db_rotations now counts only real dbengine rotations, as the guard
  expects.
@stelfrag
stelfrag requested a review from Copilot May 7, 2026 15:11

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 5 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant ChartClean as Chart Cleanup Worker
    participant Service as svc_rrd_cleanup_obsolete_charts_from_all_hosts
    participant HostClean as svc_rrdhost_cleanup_charts_marked_obsolete
    participant DimArchive as svc_rrdset_archive_obsolete_dimensions
    participant RRDSet as RRDSET
    participant RRDContext as rrdcontext.c
    participant GCWorker as RRDContext GC Worker
    participant ExtremeCard as extreme_cardinality guard

    Note over ChartClean,ExtremeCard: Chart Cleanup → Deep GC Flow

    loop Every 10s
        ChartClean->>Service: Timer fires
        Service->>Service: rrd_rdlock()
        
        loop For each host (non-replicating)
            Service->>HostClean: svc_rrdhost_cleanup_charts_marked_obsolete(host)
            
            alt Host has pending obsolete charts/dimensions
                HostClean->>HostClean: Iterate RRDSETs
                
                alt Chart has RRDSET_FLAG_OBSOLETE_DIMENSIONS
                    HostClean->>DimArchive: svc_rrdset_archive_obsolete_dimensions(st, all=false)
                    DimArchive->>DimArchive: Scan dimensions, archive obsolete
                    DimArchive->>RRDSet: Archive RRDMETRIC for each dimension
                    DimArchive-->>HostClean: Return count of archived dimensions
                    HostClean->>HostClean: archived_items += count
                end
                
                alt Chart has RRDSET_FLAG_OBSOLETE (full chart)
                    HostClean->>DimArchive: svc_rrdset_archive_obsolete_dimensions(st, all=true)
                    DimArchive->>RRDSet: Archive all dims (archived RRDMETRICs)
                    DimArchive-->>HostClean: Return count
                    alt All dimensions archived
                        HostClean->>HostClean: archived_items += count + 1 (RRDINSTANCE)
                        HostClean->>RRDSet: rrdset_free(st)
                    end
                end
                
                HostClean-->>Service: Return total archived_items
            else No pending work
                HostClean-->>Service: Return 0
            end
        end
        
        Service->>Service: rrd_rdunlock()
        
        alt archived_items > 0
            Service->>RRDContext: rrdcontext_request_full_gc()
            
            Note over RRDContext: CAS: only arm if slot == 0
            RRDContext->>RRDContext: __atomic_compare_exchange_n(rrdcontext_next_db_rotation_ut, 0, deadline+120s)
            alt CAS succeeds (slot was zero)
                Note over RRDContext: New GC pass scheduled in 120s
            else CAS fails (slot already non-zero)
                Note over RRDContext: Existing GC pass covers this request
            end
        end
    end

    Note over GCWorker,ExtremeCard: Deep GC Execution (in parallel)

    loop Worker loop (every 10s)
        GCWorker->>GCWorker: Load rrdcontext_next_db_rotation_ut
        alt Deadline reached
            GCWorker->>GCWorker: rrdcontext_recalculate_retention_all_hosts()
            GCWorker->>GCWorker: rrdcontext_garbage_collect_for_all_hosts()
            
            alt Slot unchanged (no dbengine rotation race)
                GCWorker->>GCWorker: CAS slot back to 0
            else Slot updated by dbengine rotation
                Note over GCWorker: Keep new deadline for next pass
            end
        end
        
        alt Real dbengine rotation detected
            ExtremeCard->>ExtremeCard: extreme_cardinality.db_rotations++
            Note over ExtremeCard: Guards extreme cardinality protection
        end
    end

    Note over RRDContext,GCWorker: DBEngine Rotation (unchanged)
    RRDContext->>RRDContext: rrdcontext_db_rotation()
    RRDContext->>RRDContext: __atomic_store_n(slot, deadline+120s)
    RRDContext->>RRDContext: rrdcontext_count_db_rotation() → bumps db_rotations
Loading

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

@stelfrag
stelfrag marked this pull request as ready for review May 7, 2026 15:20
@stelfrag

stelfrag commented May 7, 2026

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented May 7, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@stelfrag I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 5 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant ChartClean as Chart Cleanup Worker
    participant Service as svc_rrd_cleanup_obsolete_charts_from_all_hosts
    participant HostClean as svc_rrdhost_cleanup_charts_marked_obsolete
    participant DimArchive as svc_rrdset_archive_obsolete_dimensions
    participant RRDSet as RRDSET
    participant RRDContext as rrdcontext.c
    participant GCWorker as RRDContext GC Worker
    participant ExtremeCard as extreme_cardinality guard

    Note over ChartClean,ExtremeCard: Chart Cleanup → Deep GC Flow

    loop Every 10s
        ChartClean->>Service: Timer fires
        Service->>Service: rrd_rdlock()
        
        loop For each host (non-replicating)
            Service->>HostClean: svc_rrdhost_cleanup_charts_marked_obsolete(host)
            
            alt Host has pending obsolete charts/dimensions
                HostClean->>HostClean: Iterate RRDSETs
                
                alt Chart has RRDSET_FLAG_OBSOLETE_DIMENSIONS
                    HostClean->>DimArchive: svc_rrdset_archive_obsolete_dimensions(st, all=false)
                    DimArchive->>DimArchive: Scan dimensions, archive obsolete
                    DimArchive->>RRDSet: Archive RRDMETRIC for each dimension
                    DimArchive-->>HostClean: Return count of archived dimensions
                    HostClean->>HostClean: archived_items += count
                end
                
                alt Chart has RRDSET_FLAG_OBSOLETE (full chart)
                    HostClean->>DimArchive: svc_rrdset_archive_obsolete_dimensions(st, all=true)
                    DimArchive->>RRDSet: Archive all dims (archived RRDMETRICs)
                    DimArchive-->>HostClean: Return count
                    alt All dimensions archived
                        HostClean->>HostClean: archived_items += count + 1 (RRDINSTANCE)
                        HostClean->>RRDSet: rrdset_free(st)
                    end
                end
                
                HostClean-->>Service: Return total archived_items
            else No pending work
                HostClean-->>Service: Return 0
            end
        end
        
        Service->>Service: rrd_rdunlock()
        
        alt archived_items > 0
            Service->>RRDContext: rrdcontext_request_full_gc()
            
            Note over RRDContext: CAS: only arm if slot == 0
            RRDContext->>RRDContext: __atomic_compare_exchange_n(rrdcontext_next_db_rotation_ut, 0, deadline+120s)
            alt CAS succeeds (slot was zero)
                Note over RRDContext: New GC pass scheduled in 120s
            else CAS fails (slot already non-zero)
                Note over RRDContext: Existing GC pass covers this request
            end
        end
    end

    Note over GCWorker,ExtremeCard: Deep GC Execution (in parallel)

    loop Worker loop (every 10s)
        GCWorker->>GCWorker: Load rrdcontext_next_db_rotation_ut
        alt Deadline reached
            GCWorker->>GCWorker: rrdcontext_recalculate_retention_all_hosts()
            GCWorker->>GCWorker: rrdcontext_garbage_collect_for_all_hosts()
            
            alt Slot unchanged (no dbengine rotation race)
                GCWorker->>GCWorker: CAS slot back to 0
            else Slot updated by dbengine rotation
                Note over GCWorker: Keep new deadline for next pass
            end
        end
        
        alt Real dbengine rotation detected
            ExtremeCard->>ExtremeCard: extreme_cardinality.db_rotations++
            Note over ExtremeCard: Guards extreme cardinality protection
        end
    end

    Note over RRDContext,GCWorker: DBEngine Rotation (unchanged)
    RRDContext->>RRDContext: rrdcontext_db_rotation()
    RRDContext->>RRDContext: __atomic_store_n(slot, deadline+120s)
    RRDContext->>RRDContext: rrdcontext_count_db_rotation() → bumps db_rotations
Loading

@stelfrag
stelfrag marked this pull request as draft May 7, 2026 15:25
…ations

  Two follow-ups:

    - rrdcontext_request_full_gc() previously dropped requests whose CAS
      failed because the worker was already mid-pass with an expired
      deadline. The host whose archive triggered the request may have
      been walked already, leaving the just-archived metadata stranded
      on quiet non-dbengine hosts.

      Add a rrdcontext_full_gc_rerun_requested flag, set when the CAS
      fails AND the observed deadline is already in the past (mid-pass
      case). The worker reads-and-clears the flag at end of pass and
      arms a fresh deadline iff the slot is currently 0; concurrent
      dbengine_rotation stores survive the follow-up CAS. Future-armed
      deadlines don't set the flag because their upcoming pass will see
      the archive.

    - extreme_cardinality.db_rotations is now atomically incremented
      from rrdcontext_count_db_rotation(). The remaining read in
      rrdcontext_post_process_updates() was a plain load, mixing atomic
      writes with non-atomic reads (a C data race; TSAN would flag it).
      Switch to __atomic_load_n with RELAXED ordering.
@stelfrag
stelfrag requested a review from Copilot May 7, 2026 15:32
@sonarqubecloud

sonarqubecloud Bot commented May 7, 2026

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Comment thread src/database/contexts/rrdcontext.c
Comment thread src/database/contexts/rrdcontext-worker.c

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

@stelfrag
stelfrag marked this pull request as ready for review May 8, 2026 06:32

@thiagoftsm thiagoftsm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR is running as expected on hoss using db modes dbengine and ram. LGTM!

@stelfrag
stelfrag merged commit 11c8088 into netdata:master May 8, 2026
169 checks passed
@stelfrag
stelfrag deleted the rrdcontext-non-dbengine-gc branch May 9, 2026 13:46
@stelfrag stelfrag mentioned this pull request Jun 22, 2026
Ferroin pushed a commit that referenced this pull request Jul 15, 2026
* Fix rrdcontext metadata leak on non-dbengine hosts

  contexts: trigger deep rrdcontext GC from chart-cleanup,
  not only dbengine

    The deep rrdcontext GC that drops archived RRDINSTANCE
    / RRDMETRIC entries was scheduled only by dbengine
    rotation. On RAM-mode (and any non-dbengine) hosts that
    schedule never fires, so archived metadata accumulates
    with every chart churn -- e.g. k8s cgroup spawn/teardown.

    Add rrdcontext_request_full_gc() and invoke it from
    svc_rrd_cleanup_obsolete_charts_from_all_hosts when at
    least one chart was actually freed. Reuses the existing
    120 s debouncer slot, so dbengine hosts are unchanged.

* contexts: keep GC deadline armed and trigger on any archival

    - rrdcontext_request_full_gc() rewrote rrdcontext_next_db_rotation_ut
      on every call. Under continuous churn (10 s maintenance loop) the
      deadline kept getting pushed out by another 120 s and never
      converged, so the deep GC never fired. Only arm when the slot is 0;
      the worker zeroes it after running, at which point the next
      archival arms a fresh window.

    - svc_rrdhost_cleanup_charts_marked_obsolete returned only the
      chart-free count. The partial-archive path
      (svc_rrdset_archive_obsolete_dimensions -> rrddim_free) archives
      dimensions without freeing the surrounding chart and still produces
      archived RRDMETRIC entries that need GC. Return
      partial_archives + full_archives and trigger on either.

* contexts: count actual archives and atomicize the GC schedule slot

    - svc_rrdset_archive_obsolete_dimensions() now returns the number of
      dimensions actually archived this call, not a "fully done" bool.
      Callers detect "all candidates archived" by checking
      RRDSET_FLAG_OBSOLETE_DIMENSIONS after the call, which is cleared on
      entry and only re-set when some candidate could not be archived.

      Old contract missed partial-archive cases: archiving 3 of 5 dims
      returned false, callers stayed at 0 partial_archives, and the deep
      rrdcontext GC trigger never fired even though 3 RRDMETRICs were
      already archived.

      svc_rrdhost_cleanup_charts_marked_obsolete() now accumulates
      archived_items (every dim archived + 1 per chart freed for the
      RRDINSTANCE) and returns that as the trigger signal.

    - rrdcontext_next_db_rotation_ut is read/written from three threads
      (dbengine rotation, service maintenance, rrdcontext worker). All
      four access sites now go through __atomic_* with RELAXED ordering,
      so 32-bit platforms cannot tear the 64-bit value and the
      request_full_gc() check-then-set is a real CAS rather than a
      TOCTOU race. RELAXED is sufficient because the slot is a wake hint
      and does not publish any data the worker consumes; the worker
      re-reads dictionaries under their own locks.

* contexts: don't drop GC requests racing the worker's clear pass

  Two paths in rrdcontext_main wrote rrdcontext_next_db_rotation_ut to 0
  unconditionally: a pre-clear at the start of
  rrdcontext_recalculate_retention_all_hosts() and a final store after
  rrdcontext_garbage_collect_for_all_hosts(). With the new
  rrdcontext_request_full_gc() CAS'ing 0 -> deadline, a request landing
  between the pre-clear and the final store would arm a fresh deadline
  that the final unconditional store would silently overwrite, dropping
  the request and stranding the just-archived metadata until another
  chart-free triggered a new pass.

    - Remove the pre-clear in rrdcontext_recalculate_retention_all_hosts:
      leave the slot at the deadline value during the pass so concurrent
      request_full_gc() calls coalesce into the in-flight pass via their
      CAS-fail-on-non-zero semantics.

    - Replace the final unconditional store-zero with a CAS expecting
      the deadline we observed at the top of the branch. If a concurrent
      rrdcontext_db_rotation() (unconditional store) has armed a new
      deadline during the pass, leave it in place so the next iteration
      fires for whatever was archived after we walked.

* Address review comments

* Address review comments

* contexts: keep extreme-cardinality guard on real dbengine rotations only

  The deep-GC pass is now reachable from two trigger paths -- dbengine
  rotation and chart-cleanup -- but extreme_cardinality.db_rotations was
  incremented unconditionally inside the worker's combined branch. That
  made the cardinality-protection guard
  (`extreme_cardinality.db_rotations && ...`) activate as soon as any
  chart-cleanup pass ran, even on hosts that never rotated dbengine,
  which broke its original "wait for first rotation" semantics.

  Move the increment out of the worker and into rrdcontext_db_rotation()
  via a small rrdcontext_count_db_rotation() helper. The chart-cleanup
  trigger (rrdcontext_request_full_gc) deliberately does not call it, so
  db_rotations now counts only real dbengine rotations, as the guard
  expects.

* contexts: don't drop GC requests landing mid-pass; atomic-load db_rotations

  Two follow-ups:

    - rrdcontext_request_full_gc() previously dropped requests whose CAS
      failed because the worker was already mid-pass with an expired
      deadline. The host whose archive triggered the request may have
      been walked already, leaving the just-archived metadata stranded
      on quiet non-dbengine hosts.

      Add a rrdcontext_full_gc_rerun_requested flag, set when the CAS
      fails AND the observed deadline is already in the past (mid-pass
      case). The worker reads-and-clears the flag at end of pass and
      arms a fresh deadline iff the slot is currently 0; concurrent
      dbengine_rotation stores survive the follow-up CAS. Future-armed
      deadlines don't set the flag because their upcoming pass will see
      the archive.

    - extreme_cardinality.db_rotations is now atomically incremented
      from rrdcontext_count_db_rotation(). The remaining read in
      rrdcontext_post_process_updates() was a plain load, mixing atomic
      writes with non-atomic reads (a C data race; TSAN would flag it).
      Switch to __atomic_load_n with RELAXED ordering.

(cherry picked from commit 11c8088)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants