Skip to content

Fix memory-safety and correctness bugs surfaced by Coverity audit - #22234

Closed
ktsaou wants to merge 48 commits into
netdata:masterfrom
ktsaou:memory-corruption-hunting
Closed

Fix memory-safety and correctness bugs surfaced by Coverity audit#22234
ktsaou wants to merge 48 commits into
netdata:masterfrom
ktsaou:memory-corruption-hunting

Conversation

@ktsaou

@ktsaou ktsaou commented Apr 20, 2026

Copy link
Copy Markdown
Member

Summary

17 commits fixing memory-safety and correctness bugs surfaced by a multi-model triage of 104 outstanding Coverity CIDs on the Netdata codebase. Each commit is tied to a single Coverity defect class and represents a minimum root-cause change. Two commits are orphan findings (not Coverity-flagged) that the review surfaced.

Memory corruption / OOB

  • c1340023a4 spawn_server: reject oversized unix socket paths — stack buffer overflow via strcpy(server_addr.sun_path, path) when NETDATA_RUN_DIR is oversized. CIDs 439982, 439997.
  • 8dde0c5434 libnetdata: grow cpuset cpu parser buffer — static buffer pinned at 101 bytes on startup (caller passes system_cpus=0), causing silent truncation and OOB read on long cpuset.cpus. CID 425864.
  • 74c3baa3fb log2journal: handle trailing logfmt escapes safely — 1-byte OOB read when a logfmt line ends in a lone backslash. CID 410217.
  • fcddb9ffc1 log2journal: fix utf named-group key indexing — sign-extended index into 256-byte table on signed-char platforms with UTF-8 named groups. CID 410102.
  • 40d87845d0 health: fix malformed ${label:...} parsingsize_t underflow → OOB write on unterminated \${label: in health config. CID 439996.

Use-after-release / logic

  • c78aea03e8 dictionary: retry view inserts after stale entry cleanup — retry loop failed to reset item = NULL on rejected acquire, leaving dangling pointers reachable. Added regression test. CID 414657.
  • 8da4e70c19 worker_utilization: handle JudyHSDel failureworker_unregister() freed the workname even when JudyHSDel returned JERR, leaving a dangling pointer still indexed in the JudyHS table. CID 468190.
  • 848f07e271 daemon: remove machine guid access precheck — TOCTOU in machine_guid_get_or_create() running pre-privilege-drop. CID 457948.

Reachable crashes

  • c8a53fddef claim: guard missing 422 errorMsgKey — null-deref on strcmp(error_key, ...) when the Cloud server's 422 body omits errorMsgKey. CID 501624.
  • de07ba77b9 claim: terminate public key read buffer — missing NUL terminator on full-buffer public.pem read; subsequent JSON consumer reads past the stack buffer. CID 501623.
  • 79678f6b1d claim: handle curl option setup failures — silent curl_easy_setopt failures on admin-controlled proxy config. CID 501625.
  • a32c726277 xenstat: skip domains without libxl domain info — uninit uuid read when libxl_domain_info() fails. CID 413854.

Resource leaks

  • 1f2d93c69c claim: free split-file claim buffers — token/rooms leak on the success path of claim_agent_from_split_files(). CIDs 442342, 442346.
  • b5c505c444 systemd-journal: fix duplicate scan dir handle leakDIR* leak when nd_journal_directory_scan_recursively() early-returns on a repeated directory. CID 459644.

Other

  • bafc5ab147 datetime: fix rfc3339 fractional scaling — integer overflow in print_fraction() when callers request 7–9 digits; added regression tests. CID 457729.
  • 63b3c5d7bc sqlite: clamp alert non_clear_duration for aclkint64uint32 narrowing on SQLite column read. CID 440042.

Orphan findings (not Coverity-flagged, surfaced during review)

  • 5c5696c309 systemd-journal: pass depth+1 to recursive directory scandepth++ post-increment passed old depth to the recursive call, silently truncating deep journal hierarchies.
  • 0333f913a7 claim: cap claim response body size — unbounded HTTP response buffer → OOM DoS from a malicious Cloud endpoint. 10 MiB cap via CURLOPT_MAXFILESIZE_LARGE + write-callback guard.

Test plan

  • cmake --build build --parallel \$(nproc) clean on every commit
  • Each commit individually reviewed by three open-weight analyzers (glm-5.1, kimi-k2.5, qwen) plus codex plus opus
  • Coverity triage updated to Bug / Fix Submitted for each fixed CID
  • CI integration tests (to run on PR)

Follow-up (for team discussion — not in this PR)

Ten additional findings were surfaced by the review pipeline but deliberately left out to keep each commit tied to one CID. A separate REVIEW-FINDINGS.md covers:

  • statsd tag-parser dropping leading whitespace
  • cgroup v2 memory.max "max" sentinel missed due to trailing newline
  • cpuset parser hardening (latent)
  • Dictionary master-dict spin risk (pre-existing)
  • Unbounded claim API parameter lengths (authenticated-only)
  • Naming / cosmetic and dead-code cleanups

Audit methodology

Each CID was independently analysed by three open-weight models (glm-5.1, kimi-k2.5, qwen), then decided/fixed by codex, then reviewed by claude-opus. When analysts disagreed, codex broke the tie with independent reasoning; opus then verified the fix against the original CID plus Netdata idioms. Build was verified clean (exit 0) after every fix before Coverity triage.


Summary by cubic

Fixes memory-safety, race, and correctness bugs found by Coverity across core modules and adds guardrails to ML model training. Prevents OOB/UAF/leaks, missed wakeups, and improves diagnostics in claim, log2journal, systemd-journal, ebpf, statsd, sqlite, rrdhost, and more.

  • Bug Fixes
    • Memory safety/resources: close v2 journal FDs on migration failures in dbengine; free metadata cleanup list and pending UUID deletions on shutdown in sqlite; avoid UAF in sys_class_power_supply property loop; clone alerts_v2 config dictionary keys to avoid stack-escape UAF.
    • Concurrency/races: make ebpf module state reads/writes atomic and start function threads outside the cleanup lock; atomically load/store ACLK pending-request cancel flag; snapshot rrdhost receiver status under its lock; protect statsd histogram/timer sample buffer (reset/grow/append/flush) with its mutex; fix missed worker wakeups and use atomic workers_exit in functions_evloop; unlock apps.plugin mutex before early exit; narrow ML host mutex scope during detection and move stats reset under the mutex.
    • Claim robustness: NUL-terminate public.pem reads; handle missing errorMsgKey; check curl_easy_setopt and curl_slist_append failures; cap response size to 10 MiB and guard size * nmemb overflow in the write-callback (plus CURLOPT_MAXFILESIZE_LARGE); improve failure messages.
    • Logic/UAF: fix dictionary view insert retry after stale cleanup; only free worker_utilization workname on successful JudyHSDel and zero-init JError_t for logging; remove machine-guid TOCTOU with mkdir(..., EEXIST) and verify the path is a directory; free removed stale nodes outside the global RRD lock.
    • Crash avoidance: in xenstat, skip domains when libxl_domain_info() fails.
    • Leaks and traversal: free claim_agent_from_split_files() buffers on success; in systemd-journal, close duplicate DIR*, also close on dictionary_set() NULL, log visited-directory tracking failures, and recurse with depth + 1.
    • Correctness/diagnostics: fix proc mdstat obsolete-chart lookup so long md names are obsoleted correctly; fix RFC3339 fractional scaling for 1–9 digits and add round-trip tests; clamp SQLite non_clear_duration to uint32_t; fix deadly-signal log to include the correct thread id; reject undersized ML k-means training output and zero-initialize inlined centers to avoid publishing empty/indeterminate models.
    • Memory safety: reject oversize AF_UNIX paths and detect snprintf truncation in spawn_server; fix logfmt trailing backslash parsing and UTF named-group key indexing in log2journal; guard malformed ${label:...} in health; grow cpuset buffer in os_read_cpuset_cpus.
    • Tests: add regression cases for logfmt trailing backslash, RFC3339 formatter (format and parse) round-trips, dictionary view stale-entry replacement, and ML k-means empty-source handling.

Written for commit e01be4c. Summary will update on new commits.

ktsaou added 18 commits April 19, 2026 10:25
Coverity CID 442342 (RESOURCE_LEAK): release the token and rooms
buffers returned by read_by_filename() after
claim_agent_from_split_files() finishes using them.

This normal-exit cleanup also fixes sibling CID 442346, which
reports the same leak root cause for rooms.
Coverity CID 459644 (RESOURCE_LEAK): nd_journal_directory_scan_recursively()
opened a directory before checking whether it had already been scanned.
Close the duplicate-path DIR handle on the early-return path so repeated
directories do not leak file descriptors.
nd_journal_directory_scan_recursively() used depth++ (post-increment)
when recursing into subdirectories, which passes the caller's current
depth to the recursive call and then increments the caller's local
counter across sibling iterations. Effect: the 2nd, 3rd, ... sibling
subdirectories of the same parent get inflated depths and prematurely
hit VAR_LOG_JOURNAL_MAX_DEPTH, silently truncating legitimate scans.

Use depth + 1 instead so every recursion starts exactly one level
deeper than the current frame.
Coverity CID 413854 (UNINIT): skip the current domain when `libxl_domain_info()` fails before hashing or storing its UUID. This avoids using an uninitialized `uuid` buffer on the error path in `xenstat_collect()`.
Coverity CID 501623 (CHECKED_RETURN): `send_curl_request()` read `public.pem`
into a fixed buffer without reserving space for a trailing NUL, then passed it
through the JSON string path as a C string. Read at most `sizeof(public_key) - 1`
bytes, keep the byte count, and terminate the buffer explicitly before use.
Coverity CID 501624 (FORWARD_NULL): guard the 422 errorMsgKey comparisons in send_curl_request() when the claim server omits or mis-types errorMsgKey.
Fall back to the existing generic 422 failure message instead of dereferencing a NULL string.
The claim flow buffered the full HTTP response body with no upper bound,
which allowed a hostile endpoint to force unbounded growth during the curl
transfer. Cap the response at 10 MiB in the write callback and configure
libcurl to reject oversized responses early when the size is advertised.
Coverity CID 501625 (CHECKED_RETURN): stop ignoring curl_easy_setopt() errors when preparing claim requests. Abort the request setup with a clear failure reason instead of continuing with partially applied curl options such as a rejected proxy configuration.
Coverity CID 410102 (TAINTED_SCALAR): copy_and_convert_key() cast
PCRE2 name-table bytes to unsigned instead of unsigned char, which can
turn non-ASCII UTF bytes in named groups into huge indexes on signed-char
builds. Cast through unsigned char before indexing
journal_key_characters_map.
Coverity CID 410217 (TAINTED_SCALAR): logfmt escape parsing advanced past the terminating NUL when a key or value ended with a lone backslash. Treat trailing backslashes as literal characters so the parser stays in-bounds without changing valid escape handling.
Coverity CID 440042 (INTEGER_OVERFLOW): health_alarm_log_populate() read
non_clear_duration from SQLite directly into the uint32_t ACLK field.
Clamp negative values to 0 and oversized values to UINT32_MAX before
serializing alert log entries.
Coverity CID 439996 (INTEGER_OVERFLOW): only process `${label:...}` placeholders when the token is complete and ends with `}`. This prevents the label-name truncation math from indexing before the local buffer on unterminated placeholders.
Coverity CID 457729 (INTEGER_OVERFLOW): print_fraction() underflowed its loop bound when callers requested 7-9 fractional digits.
Scale microseconds down for shorter output and pad zeros for 7-9 digits so the formatter preserves the existing 1..9-digit contract without hanging.
Coverity CID 414657 (REVERSE_INULL): a rejected item-acquire path in
`dict_item_add_or_reset_value_and_acquire()` left `item` non-NULL, so the
`do/while` loop exited instead of retrying after stale view-entry cleanup.
Clear `item` before retrying and cover the stale-view replacement path in the
existing dictionary unittest.
Coverity CID 425864 (TAINTED_SCALAR): os_read_cpuset_cpus() sized its static buffer from the first caller's system_cpus argument, and startup can first call it with 0. Derive a non-zero CPU baseline when needed and grow the buffer before reading cpuset.cpus so long CPU lists are not truncated into wrong counts or out-of-bounds parsing.
Coverity CID 468190 (CHECKED_RETURN): worker_unregister() freed the last workname even when JudyHSDel() failed, which can leave a freed workname still indexed in the JudyHS table. Only free and account away the workname after a successful delete, and log unexpected JudyHS delete failures.
Coverity CID 457948 (TOCTOU): machine_guid_get_or_create() checked the
registry path with access() before calling mkdir() while startup still runs
with elevated privileges. Use mkdir() with EEXIST handling directly, matching
existing Netdata directory-creation idioms and removing the check/use race.
Coverity CID 439982 (STRING_OVERFLOW): reject AF_UNIX socket paths that do not fit in sun_path before the nofork spawn server stores or uses them. The nofork connect and bind paths now use bounded copies after that validation, fixing both copies of the same root cause.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 18 files

Confidence score: 4/5

  • This PR is likely safe to merge with minimal risk, but there is a concrete cleanup issue in src/libnetdata/dictionary/dictionary-unittest.c that should be addressed soon.
  • In the failure path, view_item2 is not released on the stale/deleted branch, which can leak an acquired reference and may affect test/runtime hygiene over time.
  • The issue is moderate-low severity (4/10) with high confidence (9/10), so it looks real but not strongly merge-blocking for core functionality.
  • Pay close attention to src/libnetdata/dictionary/dictionary-unittest.c - ensure view_item2 is released on all failure branches to avoid reference leaks.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/libnetdata/dictionary/dictionary-unittest.c">

<violation number="1" location="src/libnetdata/dictionary/dictionary-unittest.c:1027">
P2: Release `view_item2` on the failure branch too; currently the stale/deleted branch leaks an acquired reference.</violation>
</file>
Architecture diagram
sequenceDiagram
    participant Agent as Netdata Agent
    participant FS as Local Filesystem
    participant LibCURL as LibCURL (Netdata Cloud)
    participant Cloud as Netdata Cloud API
    participant JSON as JSON-C Parser

    Note over Agent,Cloud: Agent Claiming & Cloud Communication Flow

    Agent->>FS: Read public.pem key
    FS-->>Agent: key bytes
    Agent->>Agent: NEW: Terminate buffer with NUL byte

    Agent->>LibCURL: Initialize PUT request
    
    loop Configure Request
        Agent->>LibCURL: CHANGED: Set CURL options via macro
        alt NEW: Option setup fails (e.g. invalid proxy)
            LibCURL-->>Agent: Error
            Agent->>Agent: NEW: Log specific option failure and abort
        end
    end

    Agent->>LibCURL: NEW: Set CURLOPT_MAXFILESIZE_LARGE (10MiB)
    
    Agent->>LibCURL: curl_easy_perform()
    LibCURL->>Cloud: PUT /api/v1/claim
    
    Cloud-->>LibCURL: Response Stream
    
    loop Response Callback
        LibCURL->>Agent: Write response chunk
        alt NEW: Total size > 10MiB
            Agent-->>LibCURL: Return 0 (Abort)
            Agent->>Agent: Set failure: "Response exceeded limit"
        else Under limit
            Agent->>Agent: Buffer response
        end
    end

    LibCURL-->>Agent: Return Status (res)

    alt Success (CURLE_OK)
        Agent->>JSON: Parse response body
        JSON-->>Agent: Parsed Object
        
        opt HTTP Status 422
            Agent->>JSON: Get "errorMsgKey"
            alt NEW: errorMsgKey is NULL/Missing
                Agent->>Agent: Guard NULL to prevent strcmp crash
            else Key exists
                Agent->>Agent: Translate error key to log message
            end
        end
    else NEW: CURLE_FILESIZE_EXCEEDED
        Agent->>Agent: Handle OOM/DoS protection exit
    else Other Error
        Agent->>Agent: Log failure details (Proxy, Keys, Insecure flag)
    end

    Note over Agent,FS: NEW: machine-guid creation removes TOCTOU race by using mkdir() + EEXIST check instead of access() pre-check.
Loading

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread src/libnetdata/dictionary/dictionary-unittest.c
ktsaou added 12 commits April 22, 2026 19:26
Coverity CID 410087 (MISSING_LOCK): aclk_web_client_interrupt_cb() read pending_req_list.canceled from a libuv worker while cancel paths updated the same flag from the ACLK side.
Switch the cancel flag accesses to atomic load/store so query cancellation stays lock-free on the hot callback path without racing across threads.
Coverity CID 410121 (MISSING_LOCK): ml_host_detect_once() reset host->mls before taking host->mutex, while ml_host_stop() and the public readers access the same struct under that mutex. Move the reset into the existing critical section so host stats updates follow one lock discipline.
Coverity CID 410124 (MISSING_LOCK): histogram and timer samples updated
the shared buffer partly outside `m->histogram.ext->mutex` while the flush
path sorted the same buffer under that lock. Keep reset, growth, append, and
flush snapshot on the same mutex so samples are not dropped or read from a
partially synchronized buffer.
Coverity CID 425867 (MISSING_LOCK): use atomic load/store for the shared workers_exit latch in the worker event loop. This removes the unlocked cross-thread race without changing the existing mutex and condition-variable flow.
Coverity CID 439969 (BAD_CHECK_OF_WAIT_COND): the worker loop could miss a
new-job signal after failing to acquire runnable work and then sleep even
though a request was already queued. Scan for runnable jobs and wait under the
same mutex so queued requests cannot be stranded behind a lost wakeup.
Coverity CID 381151 (LOCK): the stored trace points to stale line numbers, but
the current print-tree exit path still called exit() while holding
apps_and_stdout_mutex. Unlock the mutex after printing and before exit() so the
module destructor does not destroy a locked uv_mutex_t.
Coverity CID 393056 (SLEEP): stop holding host->mutex across chart
dictionary traversal in ml_host_detect_once(). Aggregate the host
stats locally and publish them under the mutex so long chart-deletion
waits on the rrdset index do not block ML readers and stop paths for
the full walk.
Coverity CID 393057 (SLEEP): ml_host_stop() held host->mutex while waiting
for the host chart dictionary read lock. Reset the host stats under the
mutex and release it before walking the chart and dimension dictionaries
so lengthy chart-deletion writers do not block ML readers behind the
mutex.
Coverity CID 405089 (SLEEP): Function-triggered eBPF socket restarts held ebpf_exit_cleanup while nd_thread_create could wait and retry. Gate the new thread until state is published, so creation happens outside the cleanup mutex without racing shutdown.
Coverity CID 442107 (SLEEP): remove-stale-node held the global RRD write lock while freeing a host. Unlink the host under the lock, then run teardown/freeing outside the lock so logging and stream shutdown waits do not block global RRD progress.
Coverity CID 405474 (RESOURCE_LEAK): initialize `fd_v2` before
`nd_mmap_advanced()` and close it on both migration failure paths.
This keeps the success-path ownership transfer through
`journalfile_v2_data_set()` unchanged while preventing leaked file
descriptors when mmap setup or journal build fails.
Coverity CID 455300 (RESOURCE_LEAK): start_metadata_hosts() skipped
store_ctx_cleanup_list() once shutdown was requested, leaving a
worker-owned Judy list unreleased after ownership moved out of
metadata_event_loop(). Call the helper unconditionally so shutdown still
frees the list while its internal guard suppresses database work.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 39 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/collectors/ebpf.plugin/libbpf_api/ebpf_library.c">

<violation number="1" location="src/collectors/ebpf.plugin/libbpf_api/ebpf_library.c:530">
P2: The socket-module guard uses `!` on an enum state, so it only passes when the module is already RUNNING and skips enabling when it is NOT_RUNNING.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

network_viewer_opt.enabled = enabled;
if (enabled) {
if (!ebpf_modules[EBPF_MODULE_SOCKET_IDX].enabled)
if (!ebpf_module_enabled_get(&ebpf_modules[EBPF_MODULE_SOCKET_IDX]))

@cubic-dev-ai cubic-dev-ai Bot Apr 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The socket-module guard uses ! on an enum state, so it only passes when the module is already RUNNING and skips enabling when it is NOT_RUNNING.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/collectors/ebpf.plugin/libbpf_api/ebpf_library.c, line 530:

<comment>The socket-module guard uses `!` on an enum state, so it only passes when the module is already RUNNING and skips enabling when it is NOT_RUNNING.</comment>

<file context>
@@ -527,7 +527,7 @@ void read_collector_values(int *disable_cgroups, int update_every, netdata_ebpf_
     network_viewer_opt.enabled = enabled;
     if (enabled) {
-        if (!ebpf_modules[EBPF_MODULE_SOCKET_IDX].enabled)
+        if (!ebpf_module_enabled_get(&ebpf_modules[EBPF_MODULE_SOCKET_IDX]))
             ebpf_enable_chart(EBPF_MODULE_SOCKET_IDX, *disable_cgroups);
 
</file context>
Suggested change
if (!ebpf_module_enabled_get(&ebpf_modules[EBPF_MODULE_SOCKET_IDX]))
if (ebpf_module_enabled_get(&ebpf_modules[EBPF_MODULE_SOCKET_IDX]) == NETDATA_THREAD_EBPF_NOT_RUNNING)
Fix with Cubic

ktsaou added 2 commits April 23, 2026 15:54
Coverity CID 471887 (RESOURCE_LEAK): metadata workers transferred
pending_uuid_deletion out of the event loop, but skipped
do_pending_uuid_deletion() after shutdown started. Always run the helper
so shutdown still frees the Judy list and queued UUIDs, while keeping
metadata cleanup disabled during shutdown.
Coverity CID 451560 (UNINIT): stop publishing a model when preprocessing yields fewer than two training vectors, because k-means returns with no cluster centers in that case. Also value-initialize inlined cluster centers so empty-source conversions stay deterministic instead of carrying indeterminate dlib matrix bytes.
@sonarqubecloud

Copy link
Copy Markdown

@ktsaou

ktsaou commented Apr 27, 2026

Copy link
Copy Markdown
Member Author

Closing this in favor of the per-area split PRs prepared by @stelfrag.

All 48 commits from this branch are tracked across 8 reviewable PRs:

Coverage was verified by patch-id (with subject-based fallback for rebased adaptations); 47 commits matched a cherry-pick in one of the 7 earlier PRs, and the remaining one was opened as part 8.

@ktsaou ktsaou closed this Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants