Skip to content

Refactor function progress: move timeout and progress authority to the plugin runtime and agent - #21723

Merged
vkalintiris merged 5 commits into
netdata:masterfrom
vkalintiris:sjr-function-progress
Feb 8, 2026
Merged

Refactor function progress: move timeout and progress authority to the plugin runtime and agent#21723
vkalintiris merged 5 commits into
netdata:masterfrom
vkalintiris:sjr-function-progress

Conversation

@vkalintiris

@vkalintiris vkalintiris commented Feb 8, 2026 •
edited by cubic-dev-ai Bot
Loading

Copy link
Copy Markdown
Contributor
  • Move timeout authority to the agent: the agent now sends FUNCTION_CANCEL to plugins
    when transactions time out, instead of each plugin managing its own timeouts.
  • Move progress reporting to the plugin runtime: handlers update atomic done/total
    counters and the runtime emits FUNCTION_PROGRESS responses on a per-transaction
    ticker. Handlers no longer emit progress responses directly.
  • Introduce FunctionCallContext (carrying ProgressState + CancellationToken) as
    the new interface passed to on_call(), replacing the bare transaction ID. Remove
    on_cancellation() and on_progress() from the FunctionHandler trait.
  • Spawn a dedicated stdout writer task so outbound I/O never blocks the stdin select loop.
  • Remove the foundation crate (Timeout is no longer needed).

Additional fixes included:

  • Add /run/log/journal to default journal paths.
  • Reduce logging verbosity at info level.
  • Move the keep-alive message to the plugin runtime.

Summary by cubic

Moves timeout authority to the agent and centralizes function progress in the plugin runtime so handlers only update counters and react to cancellation. Also updates the protocol and journal engine for cancellation/progress, removes the foundation crate, adds a non-blocking writer with built‑in keepalive, and fixes packaging to ship the journal-viewer config.

  • Refactors

    • Runtime: atomic ProgressState and a per-transaction ticker that emits FUNCTION_PROGRESS (done/total); dedicated stdout writer task plus built-in keepalive.
    • Handler API: introduces FunctionCallContext (progress + CancellationToken); removes on_cancellation/on_progress from FunctionHandler.
    • Protocol: splits FUNCTION_PROGRESS into request (agent→plugin) and response (plugin→agent); direction-aware parsing and response encoding.
    • Journal engine: replaces Timeout with CancellationToken; optional progress counters in indexing and LogQuery; TimeBudgetExceeded renamed to Cancelled.
    • Journal viewer: adapts to new context; two-phase progress (indexing + query); query moved to spawn_blocking with cancellation; less info-level noise; adds /run/log/journal to defaults.
    • Agent: sends FUNCTION_CANCEL to plugins on transaction timeout; removes foundation crate.
    • Packaging: include conf.d/journal-viewer.yaml in the journal-viewer RPM subpackage.

  • Migration

    • Update FunctionHandler: on_call now takes (ctx: FunctionCallContext, request); delete on_cancellation and on_progress.
    • Report progress via ctx.progress (set_total, done_counter, or update); do not send progress messages directly.
    • Honor cancellation with ctx.cancellation (propagate into blocking/Rayon work and check before loops).
    • Update journal-engine calls: batch_compute_file_indexes now takes (CancellationToken, IndexingLimits, Option<Arc>); handle EngineError::Cancelled; LogQuery supports with_cancellation/with_progress.
    • Remove foundation from dependencies and stop using Timeout.

Written for commit 2e4b13b. Summary will update on new commits.

@vkalintiris
Refactor function progress: move timeout authority to the agent
234c946 
The plugin runtime now owns progress reporting and the agent is the
authoritative source for transaction timeouts. Function handlers only
need to update atomic progress counters and react to cancellation — they
no longer manage their own timeouts or emit progress responses directly.

Key changes:

Plugin runtime (netdata-plugin/rt):
- Add ProgressState with atomic done/total counters that handlers can
  update from any context (async, spawn_blocking, rayon).
- Add FunctionCallContext carrying ProgressState and a CancellationToken,
  passed to on_call() instead of a bare transaction ID.
- Spawn a per-transaction ticker task that reads the atomic counters once
  per second and emits FUNCTION_PROGRESS responses to the agent.
- Spawn a dedicated writer task for stdout so outbound I/O never blocks
  the main stdin select loop.
- Remove on_cancellation() and on_progress() from the FunctionHandler
  trait; the runtime handles cancellation generically.

Protocol (netdata-plugin/protocol):
- Split FunctionProgress into FunctionProgressRequest (agent→plugin) and
  FunctionProgressResponse (plugin→agent, carries done/all counters).
- Make FUNCTION_PROGRESS parsing direction-aware: input context parses
  the request form, output context parses the response form.
- Implement encoding for FunctionProgressResponse.

Journal-engine:
- Replace Timeout with CancellationToken in batch_compute_file_indexes
  and add an optional progress_counter parameter.
- Add optional cancellation and progress support to LogQuery.
- Rename TimeBudgetExceeded error to Cancelled.

Journal-viewer-plugin (CatalogFunction):
- Adapt on_call to receive FunctionCallContext; remove on_cancellation
  and on_progress implementations.
- Set total progress to 2×file count (indexing + querying) and pass
  the done counter to both phases.
- Wrap log querying in spawn_blocking with cancellation support.
- Simplify Transaction and TransactionRegistry, removing timeout
  tracking, progress flags, and cancellation state that the runtime
  now manages.

Agent (pluginsd_functions.c):
- Send FUNCTION_CANCEL to the plugin when garbage-collecting timed-out
  transactions, so the plugin can stop in-progress work.

Remove the foundation crate (Timeout is no longer needed).
@vkalintiris
Be less verbose at info-level.
2e9a7c6
@vkalintiris
Add /run/log/journal to default paths.
aa7ded3
@vkalintiris
Move the keep-alive message to the plugin runtime.
a77701d
@vkalintiris vkalintiris self-assigned this Feb 8, 2026
@vkalintiris
vkalintiris requested a review from Ferroin as a code owner February 8, 2026 15:35
cubic-dev-ai[bot]
cubic-dev-ai Bot reviewed Feb 8, 2026
View reviewed changes

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 28 files

Confidence score: 4/5

  • This PR looks safe to merge overall; the noted issues are low-to-medium severity and unlikely to be merge-blocking.
  • tokio::time::interval() in src/crates/netdata-plugin/rt/src/lib.rs will send PLUGIN_KEEPALIVE immediately at startup, which could change expected timing behavior for the first minute.
  • LogLevel::Debug mapping to trace in src/crates/netdata-plugin/rt/src/tracing_setup.rs may produce more verbose logs than users expect when selecting debug.
  • Pay close attention to src/crates/netdata-plugin/rt/src/lib.rs and src/crates/netdata-plugin/rt/src/tracing_setup.rs - timing of keepalive and log-level mapping behavior.
Prompt for AI agents (all issues) 

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="src/crates/netdata-plugin/rt/src/lib.rs">

<violation number="1" location="src/crates/netdata-plugin/rt/src/lib.rs:828">
P2: `tokio::time::interval()` fires the first tick immediately, so a `PLUGIN_KEEPALIVE` will be sent right at startup rather than after 60 seconds. Use `interval_at` to delay the first tick, or consume the initial tick before entering the loop.</violation>
</file>

<file name="src/crates/netdata-plugin/rt/src/tracing_setup.rs">

<violation number="1" location="src/crates/netdata-plugin/rt/src/tracing_setup.rs:50">
P2: `LogLevel::Debug` now maps to the trace filter, which will emit trace-level logs when the user asked for debug. This increases verbosity beyond the configured level; keep debug mapped to "debug" unless a separate trace level exists.</violation>
</file>
Architecture diagram 
sequenceDiagram
    participant Agent as Netdata Agent
    participant RT as Plugin Runtime (Stdin Loop)
    participant WT as NEW: Stdout Writer Task
    participant Handler as Function Handler
    participant Engine as Journal Engine (Blocking/Rayon)

    Note over Agent,Engine: NEW: Function Execution & Progress Flow

    Agent->>RT: FUNCTION_CALL (transaction_id, payload)
    
    RT->>RT: NEW: Create FunctionCallContext<br/>(CancellationToken + ProgressState)
    
    RT->>Handler: CHANGED: on_call(ctx, request)
    
    activate Handler
    Handler->>Engine: Start Long Operation
    Note right of Engine: Uses ctx.progress counters<br/>& ctx.cancellation token
    
    loop Work Loop
        Engine->>Engine: CHANGED: Update Atomic Counters<br/>(done_counter.fetch_add)
    end

    loop Every 1 Second (Internal Ticker)
        RT->>RT: NEW: Read ProgressState counters
        RT->>WT: Send Progress Message
        WT->>Agent: NEW: FUNCTION_PROGRESS (done, total)
    end

    Note over Agent,Engine: NEW: Timeout / Cancellation Flow

    alt Agent Timeout Reached
        Agent->>RT: NEW: FUNCTION_CANCEL (transaction_id)
        RT->>RT: Trigger ctx.cancellation.cancel()
        Engine->>Engine: Check is_cancelled()
        Engine-->>Handler: Err(Cancelled)
        Handler-->>RT: Result (Err/Partial)
    else Success Path
        Engine-->>Handler: Success
        Handler-->>RT: Result (Success)
    end
    deactivate Handler

    RT->>WT: Send Result Message
    WT-->>Agent: FUNCTION_RESULT (transaction_id, payload)

    Note over RT,WT: NEW: Keepalive Flow
    loop Internal Interval
        RT->>WT: Send Keepalive
        WT->>Agent: PLUGIN_KEEPALIVE
    end
Loading

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread src/crates/netdata-plugin/rt/src/lib.rs
Comment thread src/crates/netdata-plugin/rt/src/tracing_setup.rs
@vkalintiris
Include journal-viewer.yaml in the plugin-journal-viewer RPM subpackage.
2e4b13b 
The config file was excluded from the main netdata package but never
added to the plugin-journal-viewer %files section, so it was missing
from installed systems.
@vkalintiris
vkalintiris requested a review from a team as a code owner February 8, 2026 15:40
@github-actions github-actions Bot added the area/packaging Packaging and operating systems support label Feb 8, 2026
@vkalintiris
vkalintiris enabled auto-merge (squash) February 8, 2026 15:52
@ilyam8
ilyam8 requested a review from Copilot February 8, 2026 17:58
Copilot AI reviewed Feb 8, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors the Netdata plugin runtime/protocol so timeout cancellation is driven by the agent, and function progress reporting is emitted centrally by the runtime (handlers update shared counters instead of sending progress messages directly). This also removes the foundation crate, adds a dedicated stdout writer task (with keepalive), and updates the journal viewer/journal engine to propagate cancellation/progress.

Changes:

  • Agent now sends FUNCTION_CANCEL on transaction timeout; plugins react via cancellation tokens.
  • Protocol splits FUNCTION_PROGRESS into request (agent→plugin) vs response (plugin→agent); runtime emits progress responses from atomic counters.
  • Runtime output is funneled through a dedicated writer task; keepalive moved into the runtime.

Reviewed changes

Copilot reviewed 28 out of 29 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
src/plugins.d/pluginsd_functions.c Send FUNCTION_CANCEL to plugin when an inflight function times out.
src/crates/netdata-plugin/types/src/lib.rs Re-export new progress request/response types.
src/crates/netdata-plugin/types/src/functions.rs Split progress message into request/response structs.
src/crates/netdata-plugin/rt/src/tracing_setup.rs Adjust log level mapping and clamp noisy crates to warn.
src/crates/netdata-plugin/rt/src/lib.rs Introduce FunctionCallContext, progress ticker, and outbound writer task/queue.
src/crates/netdata-plugin/rt/Cargo.toml Drop dependency on removed foundation crate.
src/crates/netdata-plugin/protocol/src/word_iterator.rs Add next_usize() to parse progress counters.
src/crates/netdata-plugin/protocol/src/tokio_codec.rs Encode progress responses; treat progress requests as inbound-only.
src/crates/netdata-plugin/protocol/src/message_parser.rs Direction-aware parsing for FUNCTION_PROGRESS request vs response.
src/crates/netdata-plugin/protocol/src/lib.rs Re-export renamed progress types.
src/crates/netdata-plugin/foundation/src/timeout.rs Remove Timeout implementation (crate being deleted).
src/crates/netdata-plugin/foundation/src/lib.rs Remove foundation crate exports.
src/crates/netdata-plugin/foundation/Cargo.toml Remove foundation crate manifest.
src/crates/netdata-otel/otel-plugin/src/lib.rs Remove plugin-local keepalive loop (now runtime-owned).
src/crates/netdata-log-viewer/journal-viewer-plugin/src/plugin_config.rs Add /run/log/journal to default journal paths.
src/crates/netdata-log-viewer/journal-viewer-plugin/src/main.rs Remove plugin-local keepalive; reduce info-level logging noise.
src/crates/netdata-log-viewer/journal-viewer-plugin/src/catalog.rs Adapt to new handler context; add cancellation/progress plumbing and spawn_blocking query.
src/crates/netdata-log-viewer/journal-viewer-plugin/Cargo.toml Add tokio-util (CancellationToken).
src/crates/netdata-log-viewer/journal-function/src/lib.rs Stop re-exporting Timeout (removed).
src/crates/journal-registry/src/registry/mod.rs Reduce logging verbosity; ignore non-create/remove notify events silently.
src/crates/journal-index/src/file_index.rs Reduce regex-related logging verbosity.
src/crates/journal-engine/src/logs/query.rs Add optional cancellation + progress counter support to queries.
src/crates/journal-engine/src/indexing.rs Replace timeout budgeting with cancellation token + optional progress counter.
src/crates/journal-engine/src/error.rs Rename timeout error to Cancelled.
src/crates/journal-engine/examples/index.rs Update example to use CancellationToken and new indexing signature.
src/crates/journal-engine/Cargo.toml Replace foundation dependency with tokio-util.
src/crates/Cargo.toml Remove foundation crate from workspace members/deps.
src/crates/Cargo.lock Remove foundation; add tokio-util where needed.
netdata.spec.in Package journal-viewer.yaml into the RPM subpackage.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/plugins.d/pluginsd_functions.c
Comment thread src/crates/journal-engine/src/indexing.rs
Comment thread src/crates/netdata-plugin/protocol/src/tokio_codec.rs
Comment thread src/crates/netdata-plugin/rt/src/lib.rs
Comment thread src/crates/netdata-plugin/rt/src/lib.rs
Comment thread src/crates/netdata-log-viewer/journal-viewer-plugin/src/catalog.rs
Comment thread src/crates/netdata-log-viewer/journal-viewer-plugin/src/catalog.rs
@vkalintiris

vkalintiris commented Feb 8, 2026

Copy link
Copy Markdown
Contributor Author

@ilyam8 nothing exceptional from copilot.

@vkalintiris
vkalintiris merged commit 09cbde1 into netdata:master Feb 8, 2026
153 checks passed
@vkalintiris vkalintiris mentioned this pull request Feb 8, 2026
Closed
6 tasks
vkalintiris added a commit to vkalintiris/netdata that referenced this pull request Feb 10, 2026
@vkalintiris
Revert "Refactor function progress: move timeout and progress authori…
71ef5c7 
…ty to the plugin runtime and agent (netdata#21723)"

This reverts commit 09cbde1.
vkalintiris added a commit to vkalintiris/netdata that referenced this pull request Feb 10, 2026
@vkalintiris
Revert "Refactor function progress: move timeout and progress authori…
69b77d1 
…ty to the plugin runtime and agent (netdata#21723)"

This reverts commit 09cbde1.
vkalintiris added a commit that referenced this pull request Feb 11, 2026
@vkalintiris @claude
@cubic-dev-ai @Ferroin
Reinstate systemd-journal plugin. (#21729)
1c0787b 
* Revert "Refactor function progress: move timeout and progress authority to the plugin runtime and agent (#21723)"

This reverts commit 09cbde1.

* Revert "Improve log message when deferred response is too big. (#21720)"

This reverts commit 453bc9d.

* Revert "Use info-level log for undhandled modification events. (#21719)"

This reverts commit 986a187.

* Revert "Handle fields with high-cardinality and big payloads. (#21716)"

This reverts commit 80a5a9f.

* Revert "Set status flag of active journal file to archived on shutdown. (#21707)"

This reverts commit c2cd26f.

* Revert "Make new journal viewer plugin a mandatory dependency for native packages to ensure clean upgrades. (#21701)"

This reverts commit 3ee561b.

* Revert "Document logs functionality of otel plugin (#21705)"

This reverts commit 83ac4e5.

* Revert "Update README links to correct otel-plugin paths (#21699)"

This reverts commit 088c20a.

* Revert "Add object size and bounds validation when reading journal files. (#21697)"

This reverts commit e482ca8.

* Revert "Handle transient systemd state flag changes. (#21695)"

This reverts commit 5638b80.

* Revert "Keep window manager state consistent on `mmap` failures. (#21693)"

This reverts commit c56d358.

* Revert "Always capture backtrace on panic. (#21681)"

This reverts commit 191de6f.

* Revert "Use debug level for logging of requests. (#21625)"

This reverts commit 2e0dd48.

* Revert "Assortement of journal-viewer plugin fixes. (#21612)"

This reverts commit 8f048c8.

* Revert "Fix 32-bit builds of journal-viewer plugin. (#21580)"

This reverts commit d98adc0.

* Revert "Fix response when no log entries have been found. (#21581)"

This reverts commit d0c06b8.

* Revert "Proper decompression of data object payloads. (#21578)"

This reverts commit 7f4ce41.

* Revert "bring back systemd-journal.plugin setcap (from source install) (#21569)"

This reverts commit ad6ee9c.

* Revert "Use host-prefixed log directories under containers. (#21568)"

This reverts commit 1fac565.

* Revert "Allow systemd-journal plugin if journal-viewer does not exist. (#21561)"

This reverts commit 2b180ba.

* Revert "Add sentry error reporting (#21558)"

This reverts commit ffcec41.

* Revert "OTEL logs (#21356)"

This reverts commit d0905d9.

* Add Rust cargo workspace for journal-viewer and otel plugins alongside systemd-journal.plugin

Copy the Rust crates from origin/master into src/crates/ as an independent
cargo workspace. This brings back the journal-viewer plugin, otel plugin,
and their supporting crates (journal-core, journal-engine, journal-index,
journal-registry, journal-log-writer, netdata-plugin, etc.) without
modifying the existing systemd-journal.plugin or its Rust FFI backend
under src/crates/jf/.

The outer workspace's Cargo.toml excludes the jf/ directory so the two
workspaces remain independent:
- src/crates/jf/: used exclusively by the C-based systemd-journal.plugin
- src/crates/: the new cargo workspace for journal-viewer and otel plugins

* Switch otel-plugin to build from src/crates/ workspace instead of src/crates/jf/

Split the corrosion_import_crate call so that each Rust workspace is
imported independently:
- journal_reader_ffi from src/crates/jf/ (for systemd-journal.plugin)
- otel-plugin from src/crates/ (the main cargo workspace)

Also add the otel-plugin rustflags for tracing_unstable and static builds,
and update config install paths and docs map to point to
src/crates/netdata-otel/otel-plugin/.

* Add full build and packaging support for journal-viewer-plugin

Add ENABLE_PLUGIN_JOURNAL_VIEWER as a new CMake option, independent of
the existing ENABLE_PLUGIN_SYSTEMD_JOURNAL. Both plugins coexist without
conflicts or replacements.

CMake (CMakeLists.txt):
- New ENABLE_PLUGIN_JOURNAL_VIEWER option (default: ON)
- Import journal-viewer-plugin from the main src/crates/ workspace via
  Corrosion, alongside otel-plugin
- Rustflags: --cfg=tracing_unstable, io_uring_skip_arch_check on 32-bit,
  crt-static for static builds
- Install target and config file processing for journal-viewer.yaml

Packaging:
- Debian: new netdata-plugin-journal-viewer package with postinst for
  cap_dac_read_search capability (Packaging.cmake, postinst)
- RPM: new journal-viewer subpackage with caps and file entries
  (netdata.spec.in)

Installer and build entry points:
- netdata-installer.sh: --enable/--disable-plugin-journal-viewer CLI
  flags and setcap/permissions block for the plugin binary
- packaging/installer/functions.sh: PLUGIN_JOURNAL_VIEWER CMake feature
- packaging/docker/Dockerfile: --enable-plugin-journal-viewer in build
- packaging/makeself/jobs/70-netdata-git.install.sh: enable for all
  architectures except armv6l

The plugin defaults to disabled in the installer (like otel) and is
explicitly enabled in Docker and static/makeself builds.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Rename journal-viewer-plugin to otel-signal-viewer-plugin

The journal-viewer name was confusing alongside systemd-journal.plugin
since both implied journal log viewing. Rename to otel-signal-viewer to
reflect the plugin's actual purpose: viewing OTel signals (logs now,
traces in the future).

Renamed across all layers:
- Rust crate: package name, binary name, directory, config file
- CMake: ENABLE_PLUGIN_OTEL_SIGNAL_VIEWER option, Corrosion targets,
  rustflags, install targets, config processing
- RPM: subpackage, file lists, exclusions
- Debian: package definition, postinst directory
- Installer: CLI flags (--enable/--disable-plugin-otel-signal-viewer),
  variable (ENABLE_OTEL_SIGNAL_VIEWER), permissions block
- Docker: installer flag, chmod loop
- Makeself: static build flags
- Rust source: function name, chart contexts, plugin runtime name,
  config filename references
- Documentation: map.yaml label, crate README

* Fix installer flag

* Update docs/.map/map.yaml

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

* Remove duplicate source entry

* Guard Recommends tag against old RPM on Amazon Linux 2

* Disable otel-signal-viewer plugin on Windows.

* Bring back function cancellation on timeout and useful diagnostic.

* Reference correct plugin in README.md

* Remove RUSTFLAGS from static builds.

* Update function description.

* Update otel-signal-viewer configuration file.

* Replace runtime relative path resolution with CMake template substitution.

Both the otel and otel-signal-viewer plugins used a resolve_relative_path()
helper to turn relative config paths into absolute ones at runtime, using
Netdata environment variables as the base directory.

The otel-signal-viewer plugin already had a .yaml.in template with CMake
variables for its paths, making its resolve_relative_path() call redundant
(the paths were already absolute after CMake substitution).

Unify both plugins on the CMake approach:

- Convert otel.yaml to otel.yaml.in with @configdir_POST@ and @logdir_POST@
  so paths are absolute at install time.
- Add configure_file() in CMakeLists.txt for the otel plugin config (matching
  the existing pattern used by otel-signal-viewer).
- Remove resolve_relative_path() from both plugins since all stock config
  paths are now absolute after CMake processing.

User-edited configs must use absolute paths going forward, otherwise
relative paths will be resolved relative to the plugin's runtime path.

* Address packaging review comments.

- Restore BUILD_ARCH ARG in Dockerfile for correct arch in cross-build CI.
- Deduplicate NETDATA_CMAKE_OPTIONS in makeself static build script.
- Add Obsoletes for journal-viewer in RPM otel-signal-viewer package.
- Add Replaces/Conflicts for journal-viewer in DEB otel-signal-viewer package.

* Finish restoring BUILD_ARCH for Docker images.

* Fix up dependencies for native packages properly.

In reality we actually need to use a transitional package to migrate
users back to the journal viewer instead of obsoleting the old package.

* Make the signal viewer plugin a mandatory dependency of the OTel plugin.

* Address Cubic review comments.

* Fix DEB package dependencies.

* Update src/collectors/systemd-journal.plugin/passive_journal_centralization_guide_no_encryption.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Revert "Remove RUSTFLAGS from static builds."

This reverts commit 3ff5ae3.

The per-target corrosion_add_target_rustflags() for journal_reader_ffi
does not work for static builds on Alpine/musl because Corrosion
determines native-static-libs during CMake configure time, before
target-specific rustflags are applied. Without crt-static during that
extraction, Rust reports -lgcc_s as a required native library, but
Alpine doesn't ship libgcc_s.a (only libgcc_s.so). The -static linker
flag then fails to find it.

The global RUSTFLAGS export in 70-netdata-git.install.sh is set before
CMake runs, so Corrosion's configure-time execute_process inherits it
from the environment and correctly reports -lgcc_eh (which exists as a
static archive) instead of -lgcc_s.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix static build permissions for otel plugins.

The static build post-installer was missing otel-plugin and
otel-signal-viewer-plugin from the ownership/permissions setup,
causing the netdata user to get "Permission denied" when trying
to execute them. The DEB packages handled this correctly via
their postinst scripts, but the static build equivalent in
install-or-update.sh was never updated when these plugins were
added.

* Fix missing chmod 4750 fallback for otel-signal-viewer-plugin when setcap is unavailable.

The else branch (no setcap) was missing the fallback chmod for
otel-signal-viewer-plugin, so it would lack needed privileges on
systems without setcap.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Co-authored-by: Austin S. Hemmelgarn <austin@netdata.cloud>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Ancairon pushed a commit to Ancairon/netdata that referenced this pull request Feb 12, 2026
@vkalintiris @Ancairon
Refactor function progress: move timeout and progress authority to th…
415686e 
…e plugin runtime and agent (netdata#21723)

* Refactor function progress: move timeout authority to the agent

The plugin runtime now owns progress reporting and the agent is the
authoritative source for transaction timeouts. Function handlers only
need to update atomic progress counters and react to cancellation — they
no longer manage their own timeouts or emit progress responses directly.

Key changes:

Plugin runtime (netdata-plugin/rt):
- Add ProgressState with atomic done/total counters that handlers can
  update from any context (async, spawn_blocking, rayon).
- Add FunctionCallContext carrying ProgressState and a CancellationToken,
  passed to on_call() instead of a bare transaction ID.
- Spawn a per-transaction ticker task that reads the atomic counters once
  per second and emits FUNCTION_PROGRESS responses to the agent.
- Spawn a dedicated writer task for stdout so outbound I/O never blocks
  the main stdin select loop.
- Remove on_cancellation() and on_progress() from the FunctionHandler
  trait; the runtime handles cancellation generically.

Protocol (netdata-plugin/protocol):
- Split FunctionProgress into FunctionProgressRequest (agent→plugin) and
  FunctionProgressResponse (plugin→agent, carries done/all counters).
- Make FUNCTION_PROGRESS parsing direction-aware: input context parses
  the request form, output context parses the response form.
- Implement encoding for FunctionProgressResponse.

Journal-engine:
- Replace Timeout with CancellationToken in batch_compute_file_indexes
  and add an optional progress_counter parameter.
- Add optional cancellation and progress support to LogQuery.
- Rename TimeBudgetExceeded error to Cancelled.

Journal-viewer-plugin (CatalogFunction):
- Adapt on_call to receive FunctionCallContext; remove on_cancellation
  and on_progress implementations.
- Set total progress to 2×file count (indexing + querying) and pass
  the done counter to both phases.
- Wrap log querying in spawn_blocking with cancellation support.
- Simplify Transaction and TransactionRegistry, removing timeout
  tracking, progress flags, and cancellation state that the runtime
  now manages.

Agent (pluginsd_functions.c):
- Send FUNCTION_CANCEL to the plugin when garbage-collecting timed-out
  transactions, so the plugin can stop in-progress work.

Remove the foundation crate (Timeout is no longer needed).

* Be less verbose at info-level.

* Add /run/log/journal to default paths.

* Move the keep-alive message to the plugin runtime.

* Include journal-viewer.yaml in the plugin-journal-viewer RPM subpackage.

The config file was excluded from the main netdata package but never
added to the plugin-journal-viewer %files section, so it was missing
from installed systems.
Ancairon pushed a commit to Ancairon/netdata that referenced this pull request Feb 12, 2026
@vkalintiris @claude
@cubic-dev-ai @Ferroin @Ancairon
Reinstate systemd-journal plugin. (netdata#21729)
fa5570d 
* Revert "Refactor function progress: move timeout and progress authority to the plugin runtime and agent (netdata#21723)"

This reverts commit 09cbde1.

* Revert "Improve log message when deferred response is too big. (netdata#21720)"

This reverts commit 453bc9d.

* Revert "Use info-level log for undhandled modification events. (netdata#21719)"

This reverts commit 986a187.

* Revert "Handle fields with high-cardinality and big payloads. (netdata#21716)"

This reverts commit 80a5a9f.

* Revert "Set status flag of active journal file to archived on shutdown. (netdata#21707)"

This reverts commit c2cd26f.

* Revert "Make new journal viewer plugin a mandatory dependency for native packages to ensure clean upgrades. (netdata#21701)"

This reverts commit 3ee561b.

* Revert "Document logs functionality of otel plugin (netdata#21705)"

This reverts commit 83ac4e5.

* Revert "Update README links to correct otel-plugin paths (netdata#21699)"

This reverts commit 088c20a.

* Revert "Add object size and bounds validation when reading journal files. (netdata#21697)"

This reverts commit e482ca8.

* Revert "Handle transient systemd state flag changes. (netdata#21695)"

This reverts commit 5638b80.

* Revert "Keep window manager state consistent on `mmap` failures. (netdata#21693)"

This reverts commit c56d358.

* Revert "Always capture backtrace on panic. (netdata#21681)"

This reverts commit 191de6f.

* Revert "Use debug level for logging of requests. (netdata#21625)"

This reverts commit 2e0dd48.

* Revert "Assortement of journal-viewer plugin fixes. (netdata#21612)"

This reverts commit 8f048c8.

* Revert "Fix 32-bit builds of journal-viewer plugin. (netdata#21580)"

This reverts commit d98adc0.

* Revert "Fix response when no log entries have been found. (netdata#21581)"

This reverts commit d0c06b8.

* Revert "Proper decompression of data object payloads. (netdata#21578)"

This reverts commit 7f4ce41.

* Revert "bring back systemd-journal.plugin setcap (from source install) (netdata#21569)"

This reverts commit ad6ee9c.

* Revert "Use host-prefixed log directories under containers. (netdata#21568)"

This reverts commit 1fac565.

* Revert "Allow systemd-journal plugin if journal-viewer does not exist. (netdata#21561)"

This reverts commit 2b180ba.

* Revert "Add sentry error reporting (netdata#21558)"

This reverts commit ffcec41.

* Revert "OTEL logs (netdata#21356)"

This reverts commit d0905d9.

* Add Rust cargo workspace for journal-viewer and otel plugins alongside systemd-journal.plugin

Copy the Rust crates from origin/master into src/crates/ as an independent
cargo workspace. This brings back the journal-viewer plugin, otel plugin,
and their supporting crates (journal-core, journal-engine, journal-index,
journal-registry, journal-log-writer, netdata-plugin, etc.) without
modifying the existing systemd-journal.plugin or its Rust FFI backend
under src/crates/jf/.

The outer workspace's Cargo.toml excludes the jf/ directory so the two
workspaces remain independent:
- src/crates/jf/: used exclusively by the C-based systemd-journal.plugin
- src/crates/: the new cargo workspace for journal-viewer and otel plugins

* Switch otel-plugin to build from src/crates/ workspace instead of src/crates/jf/

Split the corrosion_import_crate call so that each Rust workspace is
imported independently:
- journal_reader_ffi from src/crates/jf/ (for systemd-journal.plugin)
- otel-plugin from src/crates/ (the main cargo workspace)

Also add the otel-plugin rustflags for tracing_unstable and static builds,
and update config install paths and docs map to point to
src/crates/netdata-otel/otel-plugin/.

* Add full build and packaging support for journal-viewer-plugin

Add ENABLE_PLUGIN_JOURNAL_VIEWER as a new CMake option, independent of
the existing ENABLE_PLUGIN_SYSTEMD_JOURNAL. Both plugins coexist without
conflicts or replacements.

CMake (CMakeLists.txt):
- New ENABLE_PLUGIN_JOURNAL_VIEWER option (default: ON)
- Import journal-viewer-plugin from the main src/crates/ workspace via
  Corrosion, alongside otel-plugin
- Rustflags: --cfg=tracing_unstable, io_uring_skip_arch_check on 32-bit,
  crt-static for static builds
- Install target and config file processing for journal-viewer.yaml

Packaging:
- Debian: new netdata-plugin-journal-viewer package with postinst for
  cap_dac_read_search capability (Packaging.cmake, postinst)
- RPM: new journal-viewer subpackage with caps and file entries
  (netdata.spec.in)

Installer and build entry points:
- netdata-installer.sh: --enable/--disable-plugin-journal-viewer CLI
  flags and setcap/permissions block for the plugin binary
- packaging/installer/functions.sh: PLUGIN_JOURNAL_VIEWER CMake feature
- packaging/docker/Dockerfile: --enable-plugin-journal-viewer in build
- packaging/makeself/jobs/70-netdata-git.install.sh: enable for all
  architectures except armv6l

The plugin defaults to disabled in the installer (like otel) and is
explicitly enabled in Docker and static/makeself builds.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Rename journal-viewer-plugin to otel-signal-viewer-plugin

The journal-viewer name was confusing alongside systemd-journal.plugin
since both implied journal log viewing. Rename to otel-signal-viewer to
reflect the plugin's actual purpose: viewing OTel signals (logs now,
traces in the future).

Renamed across all layers:
- Rust crate: package name, binary name, directory, config file
- CMake: ENABLE_PLUGIN_OTEL_SIGNAL_VIEWER option, Corrosion targets,
  rustflags, install targets, config processing
- RPM: subpackage, file lists, exclusions
- Debian: package definition, postinst directory
- Installer: CLI flags (--enable/--disable-plugin-otel-signal-viewer),
  variable (ENABLE_OTEL_SIGNAL_VIEWER), permissions block
- Docker: installer flag, chmod loop
- Makeself: static build flags
- Rust source: function name, chart contexts, plugin runtime name,
  config filename references
- Documentation: map.yaml label, crate README

* Fix installer flag

* Update docs/.map/map.yaml

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

* Remove duplicate source entry

* Guard Recommends tag against old RPM on Amazon Linux 2

* Disable otel-signal-viewer plugin on Windows.

* Bring back function cancellation on timeout and useful diagnostic.

* Reference correct plugin in README.md

* Remove RUSTFLAGS from static builds.

* Update function description.

* Update otel-signal-viewer configuration file.

* Replace runtime relative path resolution with CMake template substitution.

Both the otel and otel-signal-viewer plugins used a resolve_relative_path()
helper to turn relative config paths into absolute ones at runtime, using
Netdata environment variables as the base directory.

The otel-signal-viewer plugin already had a .yaml.in template with CMake
variables for its paths, making its resolve_relative_path() call redundant
(the paths were already absolute after CMake substitution).

Unify both plugins on the CMake approach:

- Convert otel.yaml to otel.yaml.in with @configdir_POST@ and @logdir_POST@
  so paths are absolute at install time.
- Add configure_file() in CMakeLists.txt for the otel plugin config (matching
  the existing pattern used by otel-signal-viewer).
- Remove resolve_relative_path() from both plugins since all stock config
  paths are now absolute after CMake processing.

User-edited configs must use absolute paths going forward, otherwise
relative paths will be resolved relative to the plugin's runtime path.

* Address packaging review comments.

- Restore BUILD_ARCH ARG in Dockerfile for correct arch in cross-build CI.
- Deduplicate NETDATA_CMAKE_OPTIONS in makeself static build script.
- Add Obsoletes for journal-viewer in RPM otel-signal-viewer package.
- Add Replaces/Conflicts for journal-viewer in DEB otel-signal-viewer package.

* Finish restoring BUILD_ARCH for Docker images.

* Fix up dependencies for native packages properly.

In reality we actually need to use a transitional package to migrate
users back to the journal viewer instead of obsoleting the old package.

* Make the signal viewer plugin a mandatory dependency of the OTel plugin.

* Address Cubic review comments.

* Fix DEB package dependencies.

* Update src/collectors/systemd-journal.plugin/passive_journal_centralization_guide_no_encryption.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Revert "Remove RUSTFLAGS from static builds."

This reverts commit 3ff5ae3.

The per-target corrosion_add_target_rustflags() for journal_reader_ffi
does not work for static builds on Alpine/musl because Corrosion
determines native-static-libs during CMake configure time, before
target-specific rustflags are applied. Without crt-static during that
extraction, Rust reports -lgcc_s as a required native library, but
Alpine doesn't ship libgcc_s.a (only libgcc_s.so). The -static linker
flag then fails to find it.

The global RUSTFLAGS export in 70-netdata-git.install.sh is set before
CMake runs, so Corrosion's configure-time execute_process inherits it
from the environment and correctly reports -lgcc_eh (which exists as a
static archive) instead of -lgcc_s.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix static build permissions for otel plugins.

The static build post-installer was missing otel-plugin and
otel-signal-viewer-plugin from the ownership/permissions setup,
causing the netdata user to get "Permission denied" when trying
to execute them. The DEB packages handled this correctly via
their postinst scripts, but the static build equivalent in
install-or-update.sh was never updated when these plugins were
added.

* Fix missing chmod 4750 fallback for otel-signal-viewer-plugin when setcap is unavailable.

The else branch (no setcap) was missing the fallback chmod for
otel-signal-viewer-plugin, so it would lack needed privileges on
systems without setcap.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Co-authored-by: Austin S. Hemmelgarn <austin@netdata.cloud>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@cubic-dev-ai cubic-dev-ai[bot] cubic-dev-ai[bot] left review comments
@ilyam8 ilyam8 ilyam8 approved these changes
@stelfrag stelfrag Awaiting requested review from stelfrag
@Ferroin Ferroin Awaiting requested review from Ferroin Ferroin is a code owner

Assignees

@vkalintiris vkalintiris

Labels

area/packaging Packaging and operating systems support area/plugins.d collectors/systemd-journal

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@vkalintiris @ilyam8