Patch release 2.10.2 - #22204
Conversation
…roppable (netdata#22201) Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> (cherry picked from commit 30e29e5)
…r fallback (netdata#22203) (cherry picked from commit 537fd8a)
There was a problem hiding this comment.
2 issues found across 18 files
Confidence score: 3/5
- There is a concrete regression risk in
src/go/plugin/framework/functions/scheduler.go:stopAccepting()flipsaccepting = falsebut only broadcasts when drained, while goroutines can block inenqueue()waiting for space; that combination can leave waiters stuck during shutdown/full-queue paths. - The test concern in
src/go/plugin/framework/functions/manager_flow_test.gois lower severity but meaningful: checking only an early "no 499" window can miss late 499s and allow false passes, reducing confidence that the behavior is fully covered. - Given one medium-severity runtime-behavior issue (5/10, high confidence) plus a test-gap issue, this looks like some merge risk rather than a hard blocker.
- Pay close attention to
src/go/plugin/framework/functions/scheduler.goandsrc/go/plugin/framework/functions/manager_flow_test.go- potential enqueue waiter wake-up gap and incomplete late-error validation.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/go/plugin/framework/functions/manager_flow_test.go">
<violation number="1" location="src/go/plugin/framework/functions/manager_flow_test.go:181">
P2: The test only verifies "no 499" in a short early window, so it can miss a late 499 and produce a false pass.</violation>
</file>
<file name="src/go/plugin/framework/functions/scheduler.go">
<violation number="1" location="src/go/plugin/framework/functions/scheduler.go:72">
P2: `stopAccepting()` sets `accepting = false` but only broadcasts when `drainedLocked()` (i.e. `pending == 0`). Since goroutines now block inside `enqueue()` waiting for space—and that only happens when the queue is full (`pending >= maxPending`)—`drainedLocked()` will be false and no broadcast fires. Blocked enqueuers won't see `!s.accepting` until another method happens to broadcast.
For consistency with the rest of this diff (which already made broadcasts unconditional in `next`, `cancelQueued`, `complete`), `stopAccepting()` should also broadcast unconditionally.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Netdata as Netdata Agent
participant In as Stdin Reader
participant FM as Functions Manager
participant Sched as Keyed Scheduler
participant Work as Framework Worker
participant JM as JobMgr / SD
participant SNMP as SNMP Collector
participant Dev as SNMP Device
Note over Netdata, Dev: Dyncfg Request Flow (with Back-pressure)
Netdata->>In: Write FUNCTION command
In->>FM: dispatchInvocation()
FM->>Sched: NEW: enqueue(req)
activate Sched
Note right of Sched: CHANGED: Blocks if queue full (Size=1)
Sched-->>FM: admitted
deactivate Sched
In-->>Netdata: Pipe read complete
Note left of In: If Sched is full, Stdin read blocks,<br/>propagating back-pressure to Agent pipe.
Sched->>Work: next()
Work->>JM: NEW: enqueueDyncfgFunction(fn)
Note right of JM: CHANGED: Blocking channel send.<br/>No handoff timeout.
JM->>JM: dyncfgSeqExec()
Note right of JM: CHANGED: Wait-decision gate is now<br/>indefinite (no 5s timeout).
Note over Netdata, Dev: Cancellation Flow
Netdata->>In: Write FUNCTION_CANCEL
In->>FM: handleCancelEvent(uid)
alt NEW: Request is still in Sched queue
FM->>FM: Ignore cancel (allow side-effects to complete)
else Request is currently running
FM->>FM: Start fallback timer
Note over FM: Delay: 5s
FM->>FM: NEW: Silent markCancelled()
Note right of FM: Tombstones UID and advances lane.<br/>Does NOT send 499 to Netdata.
FM->>Sched: complete(key, uid)
end
Note over Netdata, Dev: SNMP Collection Flow (Periodic)
SNMP->>Dev: CHANGED: SNMP GET (MaxOIDs=20)
Note right of SNMP: Removed 32-bit Fallbacks.<br/>Requests 64-bit HC counters only.
Dev-->>SNMP: PDU Response
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
(cherry picked from commit d4e069f)
There was a problem hiding this comment.
Pull request overview
Patch release 2.10.2 focusing on collector reliability and dyncfg/function pipeline correctness by reducing SNMP request size, removing ZFS crash conditions, and making dyncfg handoff non-droppable (blocking) after removing the wait-decision timeout.
Changes:
- SNMP: lower default
MaxOIDsto 20 and make IF-MIB traffic/unicast metrics 64-bit HC only (no 32-bit fallback). - Functions/dyncfg: remove wait-decision timeout and replace “queue full” rejections with blocking enqueue/back-pressure; adjust cancel semantics and runtime metrics/tests accordingly.
- Diskspace (ZFS): guard NULL
filesystem, cache pool capacity using existingstatvfs()results, and move LXC detection to a one-time startup check.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/go/plugin/go.d/config/go.d/snmp.profiles/default/_std-if-mib.yaml | Removes 32-bit fallback OIDs from the default IF-MIB profile and makes traffic/unicast virtual metrics HC-only. |
| src/go/plugin/go.d/collector/snmp/config_schema.json | Lowers default SNMP MaxOIDs to reduce GET request size. |
| src/go/plugin/go.d/collector/snmp/collector_test.go | Updates test config defaults to match new MaxOIDs. |
| src/go/plugin/go.d/collector/snmp/collector.go | Updates collector default MaxOIDs to 20. |
| src/go/plugin/framework/functions/scheduler_test.go | Replaces queue-full rejection tests with blocking-enqueue behavior tests. |
| src/go/plugin/framework/functions/scheduler.go | Changes scheduler enqueue to block when full and adds test-only waiter introspection. |
| src/go/plugin/framework/functions/runtime_metrics_test.go | Adjusts metrics expectations after removing queue-full rejection behavior/metric. |
| src/go/plugin/framework/functions/runtime_metrics.go | Removes queue_full_total counter and associated observer. |
| src/go/plugin/framework/functions/manager_flow_test.go | Updates cancel/queue semantics tests (queued cancel ignored; fallback tombstones silently). |
| src/go/plugin/framework/functions/manager.go | Makes dispatch blocking on full scheduler, changes cancel handling, and adds silent tombstone path for cancel fallback. |
| src/go/plugin/framework/functions/README.md | Documents new queue-full blocking semantics and removes queue-full metric mention/flowchart edge. |
| src/go/plugin/agent/jobmgr/manager_process_test.go | Removes test coverage tied to the deleted wait-decision timeout behavior. |
| src/go/plugin/agent/jobmgr/manager.go | Removes wait-decision timeout configuration and timeout warning log path. |
| src/go/plugin/agent/jobmgr/dyncfg_handoff.go | Changes dyncfg handoff to blocking send (no per-function timeout/busy 503). |
| src/go/plugin/agent/jobmgr/dyncfg_collector_test.go | Removes wait timeout parameter usage in jobmgr dyncfg handler tests. |
| src/go/plugin/agent/discovery/sd/wait_decision_test.go | Updates SD wait-decision tests to assert blocking until explicit enable/disable (no timeout). |
| src/go/plugin/agent/discovery/sd/sd.go | Removes wait-decision timeout and its timeout warning log path. |
| src/go/plugin/agent/discovery/sd/dyncfg_handoff.go | Changes SD dyncfg handoff to blocking send (no per-function timeout/busy 503). |
| src/collectors/diskspace.plugin/plugin_diskspace.c | Fixes NULL filesystem crash, replaces ZFS pool capacity collection with cache-on-successful-statvfs, and performs one-time LXC detection. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // fallback timer was armed and will emit 499 + tombstone after | ||
| // cancelFallbackDelay. |
There was a problem hiding this comment.
The comment is now inaccurate: cancel fallback no longer emits a 499 (it calls markCancelled() to tombstone silently). Please update the comment to reflect the new behavior so future readers don’t assume a terminal response is sent on cancel fallback.
| // fallback timer was armed and will emit 499 + tombstone after | |
| // cancelFallbackDelay. | |
| // fallback timer was armed and will silently tombstone the invocation | |
| // after cancelFallbackDelay if the handler does not finish first. |
| diskspace_slow_thread = nd_thread_create( | ||
| "P[diskspace slow]", | ||
| NETDATA_THREAD_OPTION_DEFAULT, | ||
| diskspace_slow_worker, | ||
| &slow_worker_data); | ||
|
|
||
| // LXC detection – done once; virtualised mounts inside LXC bypass the ZFS exclusion heuristic | ||
| zfs_inside_lxc_container = is_lxcfs_proc_mounted(); | ||
|
|
There was a problem hiding this comment.
zfs_inside_lxc_container is documented as being set “once at startup”, but it’s initialized after starting diskspace_slow_worker. To avoid a startup race where the slow worker observes the default false value, initialize zfs_inside_lxc_container before creating any worker threads (or otherwise synchronize access).
| # 32-bit octets (fallback for traffic) | ||
| - { OID: 1.3.6.1.2.1.2.2.1.10, name: _ifInOctets, scale_factor: 8 } | ||
| - { OID: 1.3.6.1.2.1.2.2.1.16, name: _ifOutOctets, scale_factor: 8 } | ||
| # - { OID: 1.3.6.1.2.1.2.2.1.10, name: _ifInOctets, scale_factor: 8 } | ||
| # - { OID: 1.3.6.1.2.1.2.2.1.16, name: _ifOutOctets, scale_factor: 8 } | ||
| # 32-bit unicast packets (fallback for unicast packet rate) | ||
| - { OID: 1.3.6.1.2.1.2.2.1.11, name: _ifInUcastPkts } | ||
| - { OID: 1.3.6.1.2.1.2.2.1.17, name: _ifOutUcastPkts } | ||
| # - { OID: 1.3.6.1.2.1.2.2.1.11, name: _ifInUcastPkts } | ||
| # - { OID: 1.3.6.1.2.1.2.2.1.17, name: _ifOutUcastPkts } |
There was a problem hiding this comment.
The header comments still describe these as “fallback” 32-bit counters, but the 32-bit symbols are now commented out and the virtual metrics are 64-bit only. Please adjust the comments to avoid suggesting a fallback exists in this default profile (and/or point users to using a custom profile if they need 32-bit OIDs).
Summary
Summary by cubic
Removed the wait-decision timeout and made
dyncfghandoff blocking to prevent dropped commands and wedged pipelines. Also fixed ZFS handling indiskspace.pluginand reduced SNMP request size while preferring 64‑bit HC metrics.Bug Fixes
dyncfg(ServiceDiscovery/jobmgr) andfunctions: enqueue now blocks until accepted or shutdown (no “busy” 503s), propagates back-pressure to stdin; queued CANCEL is ignored; cancel fallback no longer emits 499 and instead tombstones silently; scheduler no longer rejects with “queue full” and thequeue_full_totalmetric/doc was removed.diskspace.plugin(ZFS): fixed nullfilesystemcrash; cache ZFS pool capacity during successful stat collection (no extra statvfs pass), perform LXC detection once at startup, and improve the dataset exclusion heuristic using the cached pool size.go.d/snmp: defaultMaxOIDslowered to 20; the default IF‑MIB profile now uses only 64‑bit HC counters for traffic and unicast packets (32‑bit fallbacks removed).Migration
queue_full_totalmetric infunctions, update checks: enqueue now blocks and that metric is removed.Written for commit 60a62bb. Summary will update on new commits.