Skip to content

Fix memory-safety and correctness bugs surfaced by Coverity audit (part 5) - #22279

Merged
stelfrag merged 6 commits into
netdata:masterfrom
stelfrag:cov_fix_part5
Apr 25, 2026
Merged

Fix memory-safety and correctness bugs surfaced by Coverity audit (part 5)#22279
stelfrag merged 6 commits into
netdata:masterfrom
stelfrag:cov_fix_part5

Conversation

@stelfrag

@stelfrag stelfrag commented Apr 25, 2026

Copy link
Copy Markdown
Collaborator
Summary

Summary by cubic

Fixes race conditions and correctness issues found by Coverity: makes eBPF module state updates thread-safe, prevents thread-start races during shutdown, fixes journal directory scan depth, and resolves a dictionary stale-view insert bug with tests. Improves shutdown reliability, avoids truncated journal discovery, and removes intermittent crashes.

  • Bug Fixes
    • eBPF: replaced direct em->enabled reads/writes with atomic helpers and updated all checks/assignments across modules; sets STOPPED via helper during cleanup; statistics and shutdown paths now read a consistent state.
    • eBPF: starts function threads outside the ebpf_exit_cleanup mutex using a parked thread that only runs after state is published, avoiding deadlocks and races during restarts and shutdown.
    • systemd-journal: passes depth + 1 when recursing (not depth++) to prevent inflated depths and truncated scans; logs failures to track visited directories and stops recursion safely.
    • dictionary: clears a rejected item pointer to retry after stale view-entry cleanup; adds a regression test for stale view replacement and fixes a missing release in the test.

Written for commit 95c0002. Summary will update on new commits.

ktsaou added 6 commits April 25, 2026 16:14
Coverity CID 414657 (REVERSE_INULL): a rejected item-acquire path in
`dict_item_add_or_reset_value_and_acquire()` left `item` non-NULL, so the
`do/while` loop exited instead of retrying after stale view-entry cleanup.
Clear `item` before retrying and cover the stale-view replacement path in the
existing dictionary unittest.
Release the acquired `view_item2` on the stale/deleted failure branch of
the view-replacement regression test, matching the release on the success
branch so the test does not leak an acquired reference.
Coverity CID 405093 (MISSING_LOCK): ebpf_module.enabled was sampled from
stats and shutdown paths while module exit code updated the same plain enum
under a different synchronization regime. Convert the live cross-thread
reads and writes to atomic helpers so those state checks stay defined
without changing the existing lock layout.
Coverity CID 405089 (SLEEP): Function-triggered eBPF socket restarts held ebpf_exit_cleanup while nd_thread_create could wait and retry. Gate the new thread until state is published, so creation happens outside the cleanup mutex without racing shutdown.
nd_journal_directory_scan_recursively() used depth++ (post-increment)
when recursing into subdirectories, which passes the caller's current
depth to the recursive call and then increments the caller's local
counter across sibling iterations. Effect: the 2nd, 3rd, ... sibling
subdirectories of the same parent get inflated depths and prematurely
hit VAR_LOG_JOURNAL_MAX_DEPTH, silently truncating legitimate scans.

Use depth + 1 instead so every recursion starts exactly one level
deeper than the current frame.
When dictionary_set() returns NULL the recursive scan still closes the
current directory and bails out, but an operator looking at truncated
journal discovery had no way to know why. Log the failure before
returning so the condition is diagnosable.
@sonarqubecloud

Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 24 files

Confidence score: 3/5

  • There are concrete regression risks in core eBPF collector paths, so this sits at moderate merge risk rather than a safe-merge state.
  • In src/collectors/ebpf.plugin/libbpf_api/ebpf_library.c, negating an enum status where RUNNING is 0 can invert behavior and incorrectly gate socket chart enablement; explicit state comparisons are needed.
  • In src/collectors/ebpf.plugin/ebpf_hardirq.c, hardirq_cleanup() returns immediately, making the cleanup branch unreachable and risking missed teardown behavior.
  • Pay close attention to src/collectors/ebpf.plugin/libbpf_api/ebpf_library.c and src/collectors/ebpf.plugin/ebpf_hardirq.c - status-check logic and cleanup control flow need correction before relying on runtime behavior.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/collectors/ebpf.plugin/libbpf_api/ebpf_library.c">

<violation number="1" location="src/collectors/ebpf.plugin/libbpf_api/ebpf_library.c:530">
P1: This check negates an enum status value, which inverts the logic because RUNNING is 0. Compare explicit states instead so the socket chart is enabled only when the module is not running.</violation>
</file>

<file name="src/collectors/ebpf.plugin/ebpf_hardirq.c">

<violation number="1" location="src/collectors/ebpf.plugin/ebpf_hardirq.c:227">
P1: This cleanup branch is unreachable because `hardirq_cleanup()` returns at the top of the function.</violation>
</file>
Architecture diagram
sequenceDiagram
    participant P as eBPF Plugin Main
    participant PT as Parked Startup Thread
    participant MT as Module Collector Thread
    participant S as Shared Module State
    participant J as Journal Service
    participant D as Dictionary View

    Note over P,S: NEW: eBPF Thread Lifecycle (Race Prevention)
    P->>PT: Create parked thread with context
    PT->>PT: Loop: tinysleep() while !ready (Atomic)
    
    P->>P: Lock(ebpf_exit_cleanup)
    alt Plugin not stopping
        P->>S: NEW: ebpf_module_enabled_set(FUNCTION_RUNNING)
        P->>PT: Set run = true, ready = true
    else Shutdown in progress
        P->>PT: Set run = false, ready = true
    end
    P->>P: Unlock
    
    PT->>PT: Wake up on ready=true
    alt run is true
        PT->>MT: Start module routine (e.g., ebpf_process_thread)
    else run is false
        PT->>PT: Self-terminate
    end

    Note over P,MT: CHANGED: eBPF Status & Monitoring
    P->>S: NEW: ebpf_module_enabled_get() (Atomic Relaxed)
    Note right of S: Avoids lock contention during stats collection

    Note over MT,S: eBPF Shutdown Path
    MT->>MT: Traps signal/exit
    MT->>P: Lock(ebpf_exit_cleanup)
    MT->>S: NEW: ebpf_module_enabled_set(STOPPED)
    MT->>P: Unlock

    Note over J: CHANGED: Journal Directory Recursion
    J->>J: Scan directory
    alt Entry is Directory
        J->>J: CHANGED: Recurse with depth + 1
        Note right of J: Prevents truncated scans/inflated depth
    end
    alt Visited Dictionary Full
        J-->>J: NEW: Log error & stop recursion
    end

    Note over D: Dictionary Stale Entry Logic
    D->>D: Attempt view entry replacement
    alt Stale/Deleted item found
        D->>D: NEW: Clear rejected pointer and retry
        Note right of D: Prevents crashes on stale view lookup
    end
Loading

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread src/collectors/ebpf.plugin/libbpf_api/ebpf_library.c
Comment thread src/collectors/ebpf.plugin/ebpf_hardirq.c
@stelfrag
stelfrag marked this pull request as ready for review April 25, 2026 13:55
@stelfrag
stelfrag requested a review from thiagoftsm as a code owner April 25, 2026 13:55
Copilot AI review requested due to automatic review settings April 25, 2026 13:55
@stelfrag
stelfrag requested a review from vkalintiris as a code owner April 25, 2026 13:55
@stelfrag
stelfrag marked this pull request as draft April 25, 2026 13:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses several memory-safety, race, and correctness issues identified during a Coverity audit, focusing on safer dictionary view replacement behavior, reliable systemd journal directory traversal, and more thread-safe eBPF module state handling during runtime/shutdown.

Changes:

  • Fix dictionary view insert retry logic by resetting the rejected item pointer, and add a regression test for stale view-item replacement after master deletion.
  • Fix systemd-journal recursive scan depth handling (depth + 1) and harden visited-directory tracking failure handling.
  • Make eBPF module enabled state transitions atomic via helper accessors and adjust shutdown/exit/function-thread paths to avoid races (including “parked start” for function threads).

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/libnetdata/dictionary/dictionary-unittest.c Adds regression coverage for replacing a stale view item after master deletion.
src/libnetdata/dictionary/dictionary-item.h Fixes retry-loop correctness by clearing item after a rejected acquire.
src/collectors/systemd-journal.plugin/systemd-journal-files.c Fixes recursion depth and improves error handling when visited-dir tracking fails.
src/collectors/ebpf.plugin/libbpf_api/ebpf_library.c Replaces direct em->enabled writes/reads with atomic helper accessors.
src/collectors/ebpf.plugin/libbpf_api/ebpf.c Uses atomic helper accessor for module state reads in stats path.
src/collectors/ebpf.plugin/ebpf_vfs.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf_sync.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf_swap.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf_softirq.c Uses atomic helper accessors for module state checks/updates during cleanup.
src/collectors/ebpf.plugin/ebpf_socket.c Uses atomic helper accessors for module state updates and function exit checks.
src/collectors/ebpf.plugin/ebpf_shm.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf_process.c Uses atomic helper accessors for module state changes on invalid state / failures / exit.
src/collectors/ebpf.plugin/ebpf_oomkill.c Uses atomic helper accessors for module state checks/updates during cleanup and disable logging.
src/collectors/ebpf.plugin/ebpf_mount.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf_mdflush.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf_hardirq.c Uses atomic helper accessors for module state checks/updates during cleanup.
src/collectors/ebpf.plugin/ebpf_functions.c Introduces “parked” thread start to publish state under ebpf_exit_cleanup before running routines.
src/collectors/ebpf.plugin/ebpf_filesystem.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf_fd.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf_disk.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf_dcstat.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf_cachestat.c Uses atomic helper accessors for module state checks/updates during exit.
src/collectors/ebpf.plugin/ebpf.h Adds atomic ebpf_module_enabled_get/set() helpers and updates validity checks to use them.
src/collectors/ebpf.plugin/ebpf.c Uses atomic helper accessors for module state checks in shutdown/stats/startup paths.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@stelfrag
stelfrag marked this pull request as ready for review April 25, 2026 15:52

@thiagoftsm thiagoftsm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything ran as expected. LGTM!

@stelfrag
stelfrag merged commit 42097e1 into netdata:master Apr 25, 2026
162 checks passed
@stelfrag
stelfrag deleted the cov_fix_part5 branch April 25, 2026 17:28
@stelfrag stelfrag mentioned this pull request Jun 22, 2026
Ferroin pushed a commit that referenced this pull request Jul 15, 2026
…rt 5) (#22279)

* dictionary: retry view inserts after stale entry cleanup

Coverity CID 414657 (REVERSE_INULL): a rejected item-acquire path in
`dict_item_add_or_reset_value_and_acquire()` left `item` non-NULL, so the
`do/while` loop exited instead of retrying after stale view-entry cleanup.
Clear `item` before retrying and cover the stale-view replacement path in the
existing dictionary unittest.

* dictionary: release view item on stale replacement branch in unittest

Release the acquired `view_item2` on the stale/deleted failure branch of
the view-replacement regression test, matching the release on the success
branch so the test does not leak an acquired reference.

* ebpf.plugin: make module state accesses atomic

Coverity CID 405093 (MISSING_LOCK): ebpf_module.enabled was sampled from
stats and shutdown paths while module exit code updated the same plain enum
under a different synchronization regime. Convert the live cross-thread
reads and writes to atomic helpers so those state checks stay defined
without changing the existing lock layout.

* ebpf: start function threads outside cleanup lock

Coverity CID 405089 (SLEEP): Function-triggered eBPF socket restarts held ebpf_exit_cleanup while nd_thread_create could wait and retry. Gate the new thread until state is published, so creation happens outside the cleanup mutex without racing shutdown.

* systemd-journal: pass depth+1 to recursive directory scan

nd_journal_directory_scan_recursively() used depth++ (post-increment)
when recursing into subdirectories, which passes the caller's current
depth to the recursive call and then increments the caller's local
counter across sibling iterations. Effect: the 2nd, 3rd, ... sibling
subdirectories of the same parent get inflated depths and prematurely
hit VAR_LOG_JOURNAL_MAX_DEPTH, silently truncating legitimate scans.

Use depth + 1 instead so every recursion starts exactly one level
deeper than the current frame.

* systemd-journal: log visited directory tracking failures

When dictionary_set() returns NULL the recursive scan still closes the
current directory and bails out, but an operator looking at truncated
journal discovery had no way to know why. Log the failure before
returning so the condition is diagnosable.

---------

Co-authored-by: Costa Tsaousis <costa@netdata.cloud>
(cherry picked from commit 42097e1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants