freeipmi.plugin: fix watchdog underflow at low system uptime - #22490
Merged
Conversation
The stale-data watchdog computes `now_monotonic_usec() - 600s` and compares it to `last_iteration_ut`. Both operands are `usec_t` (uint64_t), so when the monotonic clock has less than 600 seconds since boot, the subtraction underflows to a huge value and the comparison trivially evaluates true. The plugin self-exits, and after 11 quick restarts plugins.d permanently disables it for the remainder of the Netdata service lifetime. Reproduces every time Netdata starts within ~10 minutes of system boot. Restructure the comparison to add the threshold to the last-iteration timestamp, which cannot underflow.
|
Merged
Ferroin
pushed a commit
that referenced
this pull request
Jul 15, 2026
(cherry picked from commit 3803a32)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
The freeipmi.plugin stale-data watchdog underflows when Netdata starts within ~10 minutes of system boot, killing the plugin every cycle until plugins.d permanently disables it.
Root cause
At
src/collectors/freeipmi.plugin/freeipmi_plugin.c:2031:Both operands are
usec_t(uint64_t).IPMI_RESTART_IF_SENSORS_DONT_ITERATE_EVERY_SECONDS * USEC_PER_SEC= 600,000,000 µs. WhenCLOCK_MONOTONICis below 600 s (i.e. system uptime under 10 minutes), the unsigned subtraction wraps to ~2⁶⁴, andlast_iteration_ut < HUGEis trivially true. The plugin logs "sensors have not be collected for N seconds" (where N is just the few seconds since plugin start, not a real threshold breach), then exits.Observed behavior
Reproduced on a host that boots and starts Netdata 20 s later. Journal namespace
netdata:All 11 failed restarts land within the first 5 minutes of uptime, so plugins.d's
SERIAL_FAILURES_THRESHOLD(10) trips andplugin_set_disabled()is called. By the time uptime exceeds 600 s and the underflow would no longer trigger, the plugin will never be retried during this Netdata lifetime. Result: no IPMI sensors/voltages/fans monitored until the next Netdata restart, which on long-uptime systems happens to be the only reason this bug is not seen more often.Fix
Restructure the comparison to add the threshold to the last-iteration timestamp, which cannot underflow:
last_iteration_ut + 600_000_000cannot overflow in any realistic scenario (would require monotonic clock nearUINT64_MAX).Test plan
ipmi.sensor_fan_speed,ipmi.sensor_temperature_c,ipmi.sensor_voltagecontexts appear and stay live in Netdata.journalctl --namespace netdatadoes not contain "sensors have not be collected for N seconds" within the first 10 minutes after boot.Summary by cubic
Fixes a watchdog underflow in
freeipmi.pluginthat caused restarts and permanent disablement when Netdata starts within ~10 minutes of boot. Ensures IPMI sensors keep collecting after early-boot starts.now > last + 600sto avoid unsigned underflow when uptime < 600s.plugins.dfrom disabling the plugin during early uptime.Written for commit 1527881. Summary will update on new commits.