This package is a small, reliable polling worker for Tropibot. On a dedicated, disposable Dappnode it claims one package job at a time, resolves the requested baseline (the latest published release when omitted), reuses an already-installed target only when it matches that baseline, upgrades it to the candidate, captures bounded evidence, runs deterministic and advisory log analysis, restores the intended target state, and delivers the result back to Tropibot.
It is intentionally destructive. Never connect it to a personal or production Dappnode. The dedicated node is the safety boundary: cleanup is best effort and is not a security boundary. The harness refuses its own package and core packages, mutates only the claimed target, never mounts the Docker socket, and does not execute shell commands.
Tropibot claim → atomically persist claim → execute one package job
→ heartbeat phase/cancellation → cleanup → persist exact completion JSON
→ retry completion until Tropibot acknowledges it
The harness has no public job-submission or GitHub API. It needs outbound HTTPS access to Tropibot. Its package UI opens an embedded expert operator dashboard at / with local job history, current stage and phase timing, worker and delivery state, recovery plans, package inventory, container stabilization samples, comparison signals, analyzer findings, cleanup status, and recovery controls. Raw package logs, claim tokens, and persisted completion payloads are not exposed. The dashboard can also read the current worker-lost record from Tropibot and, after the operator confirms the old process is fenced and cleanup is complete, either release the worker or release it and requeue the package. These requests are proxied by the harness backend; the shared worker token is never exposed to browser JavaScript. GET /api/jobs supplies local history and expert diagnostics, while GET /api/coordinator/lost-job supplies the coordinator recovery state. GET /healthz is a process check; GET /readyz also checks Dappmanager tool availability and unresolved local recovery state.
For every accepted job, the deterministic controller:
- verifies Dappmanager MCP tools and target safety;
- checks whether the target is already installed and records its exact version for restoration when appropriate;
- previews the requested
baselineRef, or previews the latest published release when it was omitted, and reuses an installed package only when its version matches that resolved baseline; - waits for a stable, non-empty all-running container set;
- captures normalized details and bounded, redacted log tails;
- upgrades the baseline in place to the candidate;
- repeats stabilization and capture, compares both observations, and performs optional Nexus analysis;
- restores a pre-existing target to its recorded version; for allowlisted slow packages, restores and retains the resolved published baseline for reuse; otherwise removes only the target with
deleteVolumes: true; and fails cleanup when final inventory cannot be verified or contains unexpected dependencies.
Nexus is advisory. With no key, the harness uses the heuristic analyzer only. With a key, it sends one bounded redacted request with no tools; Nexus failures fall back safely, and Nexus can never override a deterministic failure.
Required worker settings are validated on startup:
TROPIBOT_URL=https://tropibot.example
PACKAGE_HARNESS_WORKER_ID=worker-01
PACKAGE_HARNESS_WORKER_TOKEN=<shared trusted secret>
PACKAGE_HARNESS_POLL_SECONDS=10
PACKAGE_HARNESS_HEARTBEAT_SECONDS=20
The shared bearer token is appropriate for this small trusted v1 deployment. The protocol deliberately leaves room for later per-worker credentials or mTLS, but neither is part of this version. The worker never logs the token.
Use .env.example for all timeout, Dappmanager, cleanup, and optional Nexus settings. MCP_TIMEOUT_MS bounds read-only Dappmanager calls at 30 seconds by default, while MCP_MUTATION_TIMEOUT_MS gives each install, update, or removal attempt 30 minutes by default (up to one hour). Transient transport and package-download failures are retried three times with bounded exponential backoff; tune this with MCP_MUTATION_ATTEMPTS and MCP_MUTATION_RETRY_DELAY_MS. NEXUS_TIMEOUT_MS gives advisory Nexus analysis five minutes by default. RETAIN_BASELINE_PACKAGES is a comma-separated allowlist for packages with expensive downloads: after the first successful baseline install, the harness records its expected version and immutable release reference, restores it after candidate testing, and reuses it on later runs instead of removing and downloading it again. TROPIBOT_TIMEOUT_MS bounds each coordinator request. The Dappmanager MCP token and Nexus key remain local and are never sent to Tropibot or persisted in a job record.
Rust 1.96.1 is pinned by rust-toolchain.toml. The process reads a local .env if it exists; existing process variables take precedence.
mkdir -p .data
PACKAGE_MANAGER_MODE=fake \
DATA_DIR=./.data \
TROPIBOT_URL=https://tropibot.example \
PACKAGE_HARNESS_WORKER_ID=worker-01 \
PACKAGE_HARNESS_WORKER_TOKEN=development-token \
cargo runThe fake package manager makes ordinary candidates pass. A candidate reference containing unstable simulates non-running containers and install-error simulates a failed upgrade; both paths still run cleanup.
The default info logs are a concise execution narrative while retaining searchable event, run_id, package, duration, attempt, and status fields. Log levels carry severity, so messages do not repeat it with decorative prefixes. Long MCP mutations announce every attempt and configured deadline; active jobs emit a heartbeat liveness line about once per minute; stabilization logs each bounded container-health sample; Nexus logs its model, input size, timeout, duration, and fallback error; and cleanup reports both its selected recovery action and final inventory verification. Credentials and raw package evidence are never logged. Internal startup, persistence, and heartbeat lifecycle messages are available at debug.
Set RUST_LOG=dappnode_package_harness=debug,tower_http=info for polling and persistence diagnostics. The normal package default remains info to avoid logging an empty Tropibot poll on every interval.
The worker uses the fixed v1 paths:
POST /v1/package-harness/jobs/claim
POST /v1/package-harness/jobs/{jobId}/heartbeat
POST /v1/package-harness/jobs/{jobId}/complete
Every request carries Authorization: Bearer <PACKAGE_HARNESS_WORKER_TOKEN>, Content-Type: application/json, and User-Agent: dappnode-package-harness/<version>. Protocol DTOs and their golden fixtures live under src/coordinator and tests/fixtures.
Claim input is validated before persistence or execution: runId is a bounded safe identifier, source.repository is an ASCII owner/repository name, source.headSha is a 7–64 character hexadecimal commit ID, and package names and references use the shared Dappnode parsers.
Before mutating, the worker atomically persists the full claim and opaque claim token under DATA_DIR. It persists every recovery-relevant phase and, before network delivery, the exact serialized completion JSON. Transient failures and coordinator compatibility rejections retry the same bytes with capped backoff and worker-specific jitter, so a coordinator upgrade heals delivery without a worker restart. Successful delivery clears legacy delivery-recovery markers. The worker never claims another job until completion is acknowledged as recorded or duplicate.
On restart, a pending completion is retried first. An interrupted job is never rerun: if it may have mutated the target, the worker inspects Dappmanager, performs bounded cleanup, then sends an interrupted worker-error completion. A lost claim is released locally after cleanup succeeds; if Tropibot still holds unresolved coordinator state, the next claim receives a conflict and polling stops without inventing new local recovery work. Manual recovery is reserved for failed cleanup, ambiguous local records, or a conflicting completion that proves Tropibot already holds different state for the claim.
If Tropibot has marked this worker lost, the dashboard shows the exact fenced job and its last phase. The operator may choose Cleanup done — release & retry or Cleanup done — release only. The backend calls Tropibot's authenticated POST /v1/package-harness/jobs/{jobId}/ready endpoint with explicit cleanup confirmation, displays Tropibot's retry disposition, and wakes the paused polling task so no package restart is required. The emergency operator-token abandonment endpoint remains a separate Tropibot operation.
After manually completing a failed target cleanup, acknowledge that exact run through the internal supervision API:
curl --fail-with-body \
--request POST \
--header 'content-type: application/json' \
--data '{"runId":"gh-pr-177-f33e0a37375d-f0181f51d12f8734"}' \
http://HARNESS_HOST:8080/operator/recovery/continueThe endpoint accepts only cleanup-related manual holds. It preserves the claim token and stored result, records the operator-verified cleanup, clears the persisted hold, and wakes the paused worker without a container restart.
The dashboard's local worker control remains visible for completion conflicts. It shows whether cleanup is confirmed, a completion payload is persisted, the local claim is held, and Tropibot acknowledged a result. Once cleanup is confirmed, the operator can either retry the exact persisted completion or explicitly accept Tropibot's existing result and release the local claim. The latter is destructive to the saved delivery obligation and requires typing the exact run ID. The worker remains alive and paused while waiting for either action, so no package restart is needed.
For planned package updates or host maintenance, choose Drain worker in the dashboard first. The lifecycle moves from accepting to draining; no new claim request is started, while an already claimed job is allowed to finish mutation, mandatory cleanup, and persisted completion delivery. Restart only after the dashboard reports drained and Safe to restart. Resume accepting jobs cancels or exits a drain. A claim response that crosses the drain boundary is completed safely because the v1 coordinator protocol has no claim-release operation. POST /operator/worker/drain and POST /operator/worker/resume expose the same controls and require an application/json body ({}). /readyz returns unavailable while draining or drained, but /healthz remains a process liveness check. SIGTERM also requests a drain, and the Compose service grants up to two hours for active safe work to finish.
Heartbeats run independently of package execution. A cancellation request is observed at safe phase boundaries and inside stabilization polling. It prevents a new mutating phase but never bypasses required cleanup. If Tropibot no longer recognizes the claim, the worker finishes the current operation, reconciles cleanup, and releases the obsolete local claim when the target is safe.
For real nodes set at least:
PACKAGE_MANAGER_MODE=mcp
DAPPMANAGER_MCP_URL=https://dedicated-test-dappnode.example/mcp
DAPPMANAGER_MCP_TOKEN=<token>
Dappmanager must expose all seven tools listed in src/package_manager/mod.rs, including mutation tools. Before using a new Dappmanager version, run:
TROPIBOT_URL=https://tropibot.example \
PACKAGE_HARNESS_WORKER_ID=worker-01 \
PACKAGE_HARNESS_WORKER_TOKEN=development-token \
DAPPMANAGER_MCP_URL=... \
DAPPMANAGER_MCP_TOKEN=... \
cargo run -- --mcp-smokeThis checks the MCP transport, tool inventory, and normalized package listing without running a destructive test.
cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
cargo build --releaseThe Makefile provides matching development commands. To build the container and package scaffold:
docker build -t package-harness.dnp.dappnode.eth:0.1.1 .
npx @dappnode/dappnodesdk@0.3.52 build --skip_save --skip_uploadThe runtime is non-root, keeps /data in a persistent volume, has no Docker socket or host networking, drops all Linux capabilities, and uses no-new-privileges.
More detail: docs/architecture.md and docs/result-schema.md.