Skip to content

Patch release 2.10.2 - #22204

Merged
Ferroin merged 3 commits into
netdata:v2.10from
stelfrag:patch_release
Apr 14, 2026
Merged

Patch release 2.10.2#22204
Ferroin merged 3 commits into
netdata:v2.10from
stelfrag:patch_release

Conversation

@stelfrag

@stelfrag stelfrag commented Apr 14, 2026

Copy link
Copy Markdown
Collaborator
Summary
  1. Fix ZFS bugs (diskspace.plugin) (Fix ZFS bugs (diskspace.plugin) #22188)
  2. fix(go.d/snmp): reduce default MaxOIDs to 20 and remove 32-bit counter fallback (fix(go.d/snmp): reduce default MaxOIDs to 20 and remove 32-bit counter fallback #22203)
  3. fix(go.d/dyncfg): remove wait-decision timeout and make handoff non-droppable (fix(go.d/dyncfg): remove wait-decision timeout and make handoff non-droppable #22201)

Summary by cubic

Removed the wait-decision timeout and made dyncfg handoff blocking to prevent dropped commands and wedged pipelines. Also fixed ZFS handling in diskspace.plugin and reduced SNMP request size while preferring 64‑bit HC metrics.

  • Bug Fixes

    • dyncfg (ServiceDiscovery/jobmgr) and functions: enqueue now blocks until accepted or shutdown (no “busy” 503s), propagates back-pressure to stdin; queued CANCEL is ignored; cancel fallback no longer emits 499 and instead tombstones silently; scheduler no longer rejects with “queue full” and the queue_full_total metric/doc was removed.
    • diskspace.plugin (ZFS): fixed null filesystem crash; cache ZFS pool capacity during successful stat collection (no extra statvfs pass), perform LXC detection once at startup, and improve the dataset exclusion heuristic using the cached pool size.
    • go.d/snmp: default MaxOIDs lowered to 20; the default IF‑MIB profile now uses only 64‑bit HC counters for traffic and unicast packets (32‑bit fallbacks removed).
  • Migration

    • If your devices lack 64‑bit HC counters, traffic/unicast charts from the default IF‑MIB profile will disappear; use a custom profile that includes 32‑bit OIDs if required.
    • If you monitored “queue full” 503s or the queue_full_total metric in functions, update checks: enqueue now blocks and that metric is removed.

Written for commit 60a62bb. Summary will update on new commits.

ilyam8 and others added 2 commits April 14, 2026 09:15
…roppable (netdata#22201)

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
(cherry picked from commit 30e29e5)
@github-actions github-actions Bot added area/docs area/collectors Everything related to data collection collectors/go.d area/go labels Apr 14, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 18 files

Confidence score: 3/5

  • There is a concrete regression risk in src/go/plugin/framework/functions/scheduler.go: stopAccepting() flips accepting = false but only broadcasts when drained, while goroutines can block in enqueue() waiting for space; that combination can leave waiters stuck during shutdown/full-queue paths.
  • The test concern in src/go/plugin/framework/functions/manager_flow_test.go is lower severity but meaningful: checking only an early "no 499" window can miss late 499s and allow false passes, reducing confidence that the behavior is fully covered.
  • Given one medium-severity runtime-behavior issue (5/10, high confidence) plus a test-gap issue, this looks like some merge risk rather than a hard blocker.
  • Pay close attention to src/go/plugin/framework/functions/scheduler.go and src/go/plugin/framework/functions/manager_flow_test.go - potential enqueue waiter wake-up gap and incomplete late-error validation.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/go/plugin/framework/functions/manager_flow_test.go">

<violation number="1" location="src/go/plugin/framework/functions/manager_flow_test.go:181">
P2: The test only verifies "no 499" in a short early window, so it can miss a late 499 and produce a false pass.</violation>
</file>

<file name="src/go/plugin/framework/functions/scheduler.go">

<violation number="1" location="src/go/plugin/framework/functions/scheduler.go:72">
P2: `stopAccepting()` sets `accepting = false` but only broadcasts when `drainedLocked()` (i.e. `pending == 0`). Since goroutines now block inside `enqueue()` waiting for space—and that only happens when the queue is full (`pending >= maxPending`)—`drainedLocked()` will be false and no broadcast fires. Blocked enqueuers won't see `!s.accepting` until another method happens to broadcast.

For consistency with the rest of this diff (which already made broadcasts unconditional in `next`, `cancelQueued`, `complete`), `stopAccepting()` should also broadcast unconditionally.</violation>
</file>
Architecture diagram
sequenceDiagram
    participant Netdata as Netdata Agent
    participant In as Stdin Reader
    participant FM as Functions Manager
    participant Sched as Keyed Scheduler
    participant Work as Framework Worker
    participant JM as JobMgr / SD
    participant SNMP as SNMP Collector
    participant Dev as SNMP Device

    Note over Netdata, Dev: Dyncfg Request Flow (with Back-pressure)

    Netdata->>In: Write FUNCTION command
    In->>FM: dispatchInvocation()
    
    FM->>Sched: NEW: enqueue(req)
    activate Sched
    Note right of Sched: CHANGED: Blocks if queue full (Size=1)
    Sched-->>FM: admitted
    deactivate Sched

    In-->>Netdata: Pipe read complete
    Note left of In: If Sched is full, Stdin read blocks,<br/>propagating back-pressure to Agent pipe.

    Sched->>Work: next()
    Work->>JM: NEW: enqueueDyncfgFunction(fn)
    Note right of JM: CHANGED: Blocking channel send.<br/>No handoff timeout.

    JM->>JM: dyncfgSeqExec()
    Note right of JM: CHANGED: Wait-decision gate is now<br/>indefinite (no 5s timeout).

    Note over Netdata, Dev: Cancellation Flow

    Netdata->>In: Write FUNCTION_CANCEL
    In->>FM: handleCancelEvent(uid)

    alt NEW: Request is still in Sched queue
        FM->>FM: Ignore cancel (allow side-effects to complete)
    else Request is currently running
        FM->>FM: Start fallback timer
        Note over FM: Delay: 5s
        FM->>FM: NEW: Silent markCancelled()
        Note right of FM: Tombstones UID and advances lane.<br/>Does NOT send 499 to Netdata.
        FM->>Sched: complete(key, uid)
    end

    Note over Netdata, Dev: SNMP Collection Flow (Periodic)

    SNMP->>Dev: CHANGED: SNMP GET (MaxOIDs=20)
    Note right of SNMP: Removed 32-bit Fallbacks.<br/>Requests 64-bit HC counters only.
    Dev-->>SNMP: PDU Response
Loading

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread src/go/plugin/framework/functions/manager_flow_test.go
Comment thread src/go/plugin/framework/functions/scheduler.go
@Ferroin
Ferroin marked this pull request as ready for review April 14, 2026 12:49
Copilot AI review requested due to automatic review settings April 14, 2026 12:49
@Ferroin
Ferroin merged commit a28cb58 into netdata:v2.10 Apr 14, 2026
158 of 161 checks passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Patch release 2.10.2 focusing on collector reliability and dyncfg/function pipeline correctness by reducing SNMP request size, removing ZFS crash conditions, and making dyncfg handoff non-droppable (blocking) after removing the wait-decision timeout.

Changes:

  • SNMP: lower default MaxOIDs to 20 and make IF-MIB traffic/unicast metrics 64-bit HC only (no 32-bit fallback).
  • Functions/dyncfg: remove wait-decision timeout and replace “queue full” rejections with blocking enqueue/back-pressure; adjust cancel semantics and runtime metrics/tests accordingly.
  • Diskspace (ZFS): guard NULL filesystem, cache pool capacity using existing statvfs() results, and move LXC detection to a one-time startup check.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/go/plugin/go.d/config/go.d/snmp.profiles/default/_std-if-mib.yaml Removes 32-bit fallback OIDs from the default IF-MIB profile and makes traffic/unicast virtual metrics HC-only.
src/go/plugin/go.d/collector/snmp/config_schema.json Lowers default SNMP MaxOIDs to reduce GET request size.
src/go/plugin/go.d/collector/snmp/collector_test.go Updates test config defaults to match new MaxOIDs.
src/go/plugin/go.d/collector/snmp/collector.go Updates collector default MaxOIDs to 20.
src/go/plugin/framework/functions/scheduler_test.go Replaces queue-full rejection tests with blocking-enqueue behavior tests.
src/go/plugin/framework/functions/scheduler.go Changes scheduler enqueue to block when full and adds test-only waiter introspection.
src/go/plugin/framework/functions/runtime_metrics_test.go Adjusts metrics expectations after removing queue-full rejection behavior/metric.
src/go/plugin/framework/functions/runtime_metrics.go Removes queue_full_total counter and associated observer.
src/go/plugin/framework/functions/manager_flow_test.go Updates cancel/queue semantics tests (queued cancel ignored; fallback tombstones silently).
src/go/plugin/framework/functions/manager.go Makes dispatch blocking on full scheduler, changes cancel handling, and adds silent tombstone path for cancel fallback.
src/go/plugin/framework/functions/README.md Documents new queue-full blocking semantics and removes queue-full metric mention/flowchart edge.
src/go/plugin/agent/jobmgr/manager_process_test.go Removes test coverage tied to the deleted wait-decision timeout behavior.
src/go/plugin/agent/jobmgr/manager.go Removes wait-decision timeout configuration and timeout warning log path.
src/go/plugin/agent/jobmgr/dyncfg_handoff.go Changes dyncfg handoff to blocking send (no per-function timeout/busy 503).
src/go/plugin/agent/jobmgr/dyncfg_collector_test.go Removes wait timeout parameter usage in jobmgr dyncfg handler tests.
src/go/plugin/agent/discovery/sd/wait_decision_test.go Updates SD wait-decision tests to assert blocking until explicit enable/disable (no timeout).
src/go/plugin/agent/discovery/sd/sd.go Removes wait-decision timeout and its timeout warning log path.
src/go/plugin/agent/discovery/sd/dyncfg_handoff.go Changes SD dyncfg handoff to blocking send (no per-function timeout/busy 503).
src/collectors/diskspace.plugin/plugin_diskspace.c Fixes NULL filesystem crash, replaces ZFS pool capacity collection with cache-on-successful-statvfs, and performs one-time LXC detection.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +461 to +462
// fallback timer was armed and will emit 499 + tombstone after
// cancelFallbackDelay.

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment is now inaccurate: cancel fallback no longer emits a 499 (it calls markCancelled() to tombstone silently). Please update the comment to reflect the new behavior so future readers don’t assume a terminal response is sent on cancel fallback.

Suggested change
// fallback timer was armed and will emit 499 + tombstone after
// cancelFallbackDelay.
// fallback timer was armed and will silently tombstone the invocation
// after cancelFallbackDelay if the handler does not finish first.

Copilot uses AI. Check for mistakes.
Comment on lines 1122 to +1130
diskspace_slow_thread = nd_thread_create(
"P[diskspace slow]",
NETDATA_THREAD_OPTION_DEFAULT,
diskspace_slow_worker,
&slow_worker_data);

// LXC detection – done once; virtualised mounts inside LXC bypass the ZFS exclusion heuristic
zfs_inside_lxc_container = is_lxcfs_proc_mounted();

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

zfs_inside_lxc_container is documented as being set “once at startup”, but it’s initialized after starting diskspace_slow_worker. To avoid a startup race where the slow worker observes the default false value, initialize zfs_inside_lxc_container before creating any worker threads (or otherwise synchronize access).

Copilot uses AI. Check for mistakes.
Comment on lines 22 to +27
# 32-bit octets (fallback for traffic)
- { OID: 1.3.6.1.2.1.2.2.1.10, name: _ifInOctets, scale_factor: 8 }
- { OID: 1.3.6.1.2.1.2.2.1.16, name: _ifOutOctets, scale_factor: 8 }
# - { OID: 1.3.6.1.2.1.2.2.1.10, name: _ifInOctets, scale_factor: 8 }
# - { OID: 1.3.6.1.2.1.2.2.1.16, name: _ifOutOctets, scale_factor: 8 }
# 32-bit unicast packets (fallback for unicast packet rate)
- { OID: 1.3.6.1.2.1.2.2.1.11, name: _ifInUcastPkts }
- { OID: 1.3.6.1.2.1.2.2.1.17, name: _ifOutUcastPkts }
# - { OID: 1.3.6.1.2.1.2.2.1.11, name: _ifInUcastPkts }
# - { OID: 1.3.6.1.2.1.2.2.1.17, name: _ifOutUcastPkts }

Copilot AI Apr 14, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The header comments still describe these as “fallback” 32-bit counters, but the 32-bit symbols are now commented out and the virtual metrics are 64-bit only. Please adjust the comments to avoid suggesting a fallback exists in this default profile (and/or point users to using a custom profile if they need 32-bit OIDs).

Copilot uses AI. Check for mistakes.
@stelfrag
stelfrag deleted the patch_release branch April 14, 2026 14:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants