Skip to content

freeipmi.plugin: fix watchdog underflow at low system uptime - #22490

Merged
ktsaou merged 1 commit into
masterfrom
fix/freeipmi-watchdog-underflow
May 17, 2026
Merged

freeipmi.plugin: fix watchdog underflow at low system uptime#22490
ktsaou merged 1 commit into
masterfrom
fix/freeipmi-watchdog-underflow

Conversation

@ktsaou

@ktsaou ktsaou commented May 15, 2026

Copy link
Copy Markdown
Member

Summary

The freeipmi.plugin stale-data watchdog underflows when Netdata starts within ~10 minutes of system boot, killing the plugin every cycle until plugins.d permanently disables it.

Root cause

At src/collectors/freeipmi.plugin/freeipmi_plugin.c:2031:

if(state.sensors.last_iteration_ut < now_monotonic_usec() - IPMI_RESTART_IF_SENSORS_DONT_ITERATE_EVERY_SECONDS * USEC_PER_SEC) {

Both operands are usec_t (uint64_t). IPMI_RESTART_IF_SENSORS_DONT_ITERATE_EVERY_SECONDS * USEC_PER_SEC = 600,000,000 µs. When CLOCK_MONOTONIC is below 600 s (i.e. system uptime under 10 minutes), the unsigned subtraction wraps to ~2⁶⁴, and last_iteration_ut < HUGE is trivially true. The plugin logs "sensors have not be collected for N seconds" (where N is just the few seconds since plugin start, not a real threshold breach), then exits.

Observed behavior

Reproduced on a host that boots and starts Netdata 20 s later. Journal namespace netdata:

17:02:26 freeipmi.plugin[12122]: main(): update frequency 1 seconds is too small for IPMI. Using 5.
17:02:45 freeipmi.plugin[12122]: main(): sensors have not be collected for 19 seconds. Exiting to restart.
17:02:45 netdata[3125]: PLUGINSD: freeipmi.plugin (pid 12122) disconnected after 0 successful data collections.
...
17:07:45 netdata[3125]: PLUGINSD: ... We have tried to collect something 11 times - unsuccessfully. Disabling it.

All 11 failed restarts land within the first 5 minutes of uptime, so plugins.d's SERIAL_FAILURES_THRESHOLD (10) trips and plugin_set_disabled() is called. By the time uptime exceeds 600 s and the underflow would no longer trigger, the plugin will never be retried during this Netdata lifetime. Result: no IPMI sensors/voltages/fans monitored until the next Netdata restart, which on long-uptime systems happens to be the only reason this bug is not seen more often.

Fix

Restructure the comparison to add the threshold to the last-iteration timestamp, which cannot underflow:

if(now_monotonic_usec() > state.sensors.last_iteration_ut + IPMI_RESTART_IF_SENSORS_DONT_ITERATE_EVERY_SECONDS * USEC_PER_SEC) {

last_iteration_ut + 600_000_000 cannot overflow in any realistic scenario (would require monotonic clock near UINT64_MAX).

Test plan

  • Boot a Linux host with IPMI hardware and Netdata configured to start within 10 minutes of boot (default systemd ordering does this).
  • Confirm ipmi.sensor_fan_speed, ipmi.sensor_temperature_c, ipmi.sensor_voltage contexts appear and stay live in Netdata.
  • Confirm journalctl --namespace netdata does not contain "sensors have not be collected for N seconds" within the first 10 minutes after boot.

Summary by cubic

Fixes a watchdog underflow in freeipmi.plugin that caused restarts and permanent disablement when Netdata starts within ~10 minutes of boot. Ensures IPMI sensors keep collecting after early-boot starts.

  • Bug Fixes
    • Changed the restart check to now > last + 600s to avoid unsigned underflow when uptime < 600s.
    • Stops false exits and prevents plugins.d from disabling the plugin during early uptime.

Written for commit 1527881. Summary will update on new commits.

The stale-data watchdog computes `now_monotonic_usec() - 600s` and
compares it to `last_iteration_ut`. Both operands are `usec_t`
(uint64_t), so when the monotonic clock has less than 600 seconds
since boot, the subtraction underflows to a huge value and the
comparison trivially evaluates true. The plugin self-exits, and
after 11 quick restarts plugins.d permanently disables it for the
remainder of the Netdata service lifetime.

Reproduces every time Netdata starts within ~10 minutes of system
boot. Restructure the comparison to add the threshold to the
last-iteration timestamp, which cannot underflow.
@ktsaou
ktsaou requested a review from thiagoftsm as a code owner May 15, 2026 08:29
@github-actions github-actions Bot added area/collectors Everything related to data collection collectors/freeipmi labels May 15, 2026
@sonarqubecloud

Copy link
Copy Markdown

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

@ktsaou
ktsaou merged commit 3803a32 into master May 17, 2026
151 of 157 checks passed
@ilyam8
ilyam8 deleted the fix/freeipmi-watchdog-underflow branch May 22, 2026 10:11
@stelfrag stelfrag mentioned this pull request Jun 22, 2026
Ferroin pushed a commit that referenced this pull request Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/collectors Everything related to data collection collectors/freeipmi

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant