Skip to content

Add test for ARAL race condition and fix - #22212

Merged
stelfrag merged 6 commits into
netdata:masterfrom
stelfrag:fix_aral_race_condition
Apr 16, 2026
Merged

Add test for ARAL race condition and fix#22212
stelfrag merged 6 commits into
netdata:masterfrom
stelfrag:fix_aral_race_condition

Conversation

@stelfrag

@stelfrag stelfrag commented Apr 16, 2026

Copy link
Copy Markdown
Collaborator
Summary
  • New test to detect aral allocator race condition
  • Add fix

Summary by cubic

Fixes an ARAL allocator race where a page chosen with a free slot becomes full before it’s locked by adding a lock-time retry and a refcount guard. Adds targeted unit and stress tests to reproduce the race and validate retried allocations and page list transitions.

  • Bug Fixes

    • Retry page selection if free_elements is 0 after locking; release + reacquire safely.
    • Guard aral_page_acquire to roll back and fail on non-positive refcounts.
  • New Features

    • Internal race unittest with coordination hook; improved aral_detect_acquire_to_page_lock_race and clean teardown (thread error handling, conditional join).
    • Stress test integration and clearer test output via fprintf; ensures forced-full page moves to the full list and validates allocator retry to a new page.

Written for commit 4aa380e. Summary will update on new commits.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.
Architecture diagram
sequenceDiagram
    participant T as Allocator Thread
    participant AL as ARAL Logic
    participant H as Test Hook (Internal Checks)
    participant P as ARAL Page
    participant O as Racing Thread

    T->>AL: aral_get_first_page_with_a_free_slot()
    
    rect rgb(240,240,240)
    Note over AL: NEW: retry_acquisition loop
    
    AL->>AL: aral_acquire_first_page()
    AL-->>AL: Target Page Found (Refcount++)

    opt NETDATA_INTERNAL_CHECKS enabled
        AL->>H: NEW: aral_unittest_wait_for_race_window()
        Note over H,O: Allocator thread pauses here
        O->>P: Racing thread fills last slot
        O->>P: Page moved to 'Full' list
        H-->>AL: Release paused allocator
    end

    AL->>P: aral_page_lock(page)

    alt Page has free slots (Happy Path)
        AL-->>T: Return Page for allocation
    else Page is full (Race Condition detected)
        Note right of AL: Detected via !page_lock.free_elements
        AL->>P: aral_page_unlock(page)
        AL->>P: aral_page_release(page) (Refcount--)
        AL->>AL: CHANGED: goto retry_acquisition
    end

    end
Loading

@stelfrag
stelfrag marked this pull request as ready for review April 16, 2026 10:09
Copilot AI review requested due to automatic review settings April 16, 2026 10:09
@stelfrag
stelfrag marked this pull request as draft April 16, 2026 10:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a deterministic internal-checks unit test intended to reproduce an ARAL allocator race, and updates ARAL page acquisition logic to safely retry when a page becomes full between “acquire” and “page lock”.

Changes:

  • Add NETDATA_INTERNAL_CHECKS-gated race-test hook + unit test (aral_detect_acquire_to_page_lock_race()).
  • Fix allocator race by retrying if free_elements == 0 after locking the selected page.
  • Tighten aral_page_acquire() by rolling back the refcount increment when acquisition is invalid (rf <= 0).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/libnetdata/aral/aral.c
Comment thread src/libnetdata/aral/aral.c Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/libnetdata/aral/aral.c Outdated
…and ensure consistent page state handling (unittest)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/libnetdata/aral/aral.c Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/libnetdata/aral/aral.c
Comment thread src/libnetdata/aral/aral.c Outdated
…eardown

- Add error handling for ARAL allocator thread creation.
- Prevent null thread dereference and streamline cleanup with conditional `nd_thread_join`.
@stelfrag
stelfrag marked this pull request as ready for review April 16, 2026 14:19

@thiagoftsm thiagoftsm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR is working as expected locally. LGTM!

@stelfrag
stelfrag merged commit 9f7887e into netdata:master Apr 16, 2026
150 of 152 checks passed
@stelfrag
stelfrag deleted the fix_aral_race_condition branch April 16, 2026 16:09
nedi-app Bot pushed a commit that referenced this pull request Apr 24, 2026
* Add stress test and unit test for ARAL allocator race condition fixes

* Fix race condition and cleanup tests

* Reorder `#ifdef NETDATA_INTERNAL_CHECKS` to fix struct placement and improve readability

* Refactor `aral_race_unittest_force_page_full` to return forced entry and ensure consistent page state handling (unittest)

* Fix `aral_detect_acquire_to_page_lock_race` to handle retried allocations and improve page state validation

* Improve ARAL race unittest to handle thread errors and ensure clean teardown

- Add error handling for ARAL allocator thread creation.
- Prevent null thread dereference and streamline cleanup with conditional `nd_thread_join`.
@stelfrag stelfrag mentioned this pull request Jun 22, 2026
Ferroin pushed a commit that referenced this pull request Jul 15, 2026
* Add stress test and unit test for ARAL allocator race condition fixes

* Fix race condition and cleanup tests

* Reorder `#ifdef NETDATA_INTERNAL_CHECKS` to fix struct placement and improve readability

* Refactor `aral_race_unittest_force_page_full` to return forced entry and ensure consistent page state handling (unittest)

* Fix `aral_detect_acquire_to_page_lock_race` to handle retried allocations and improve page state validation

* Improve ARAL race unittest to handle thread errors and ensure clean teardown

- Add error handling for ARAL allocator thread creation.
- Prevent null thread dereference and streamline cleanup with conditional `nd_thread_join`.

(cherry picked from commit 9f7887e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants