fix(go.d/dyncfg): remove wait-decision timeout and make handoff non-droppable - #22201

…roppable

The 5s wait-decision timer in jobmgr and SD was a false-positive race
against the single-threaded run loop: when CmdEnable's Check() took ~5s
(reproduced with time.Sleep in snmp), unrelated discovered configs would
log "timed out waiting for enable/disable decision" even though their
decision command was already queued and would be processed shortly. The
loop just couldn't read it in time. Removed the timer.

Without the timer, an awaited enable/disable that gets dropped upstream
would wedge the wait gate forever. Closed every drop point in the dyncfg
chain so commands can no longer be silently lost:

- framework/functions scheduler.enqueue now blocks on cond.Wait when full
  instead of returning errSchedulerQueueFull (was: 503 to netdata).
  next/cancelQueued/complete broadcast cond so producers wake when space
  frees. A small ctx-watcher in Manager.run calls scheduler.stop() on
  ctx.Done so blocked enqueue waiters unblock during shutdown.
- jobmgr and SD enqueueDyncfgFunction now use a plain blocking channel
  send guarded only by the base context (was: BoundedSend with per-fn
  timeout -> 503). The per-fn timeout removal is intentional: it was the
  source of late drops that could starve the wait gate.

Back-pressure now propagates upstream via the OS pipe (stdin reader pauses
when the chain is jammed) instead of being absorbed into 503 responses.

Removed the now-dead queue_full_total runtime metric and updated the
framework README to reflect the blocking admission semantics. Tests that
exercised the wait-timeout / queue-full paths were removed or rewritten.

Out of scope: netdata-core "FUNCTION_RESULT_BEGIN ... transaction is not
found" logs caused by the 120s netdata transaction timeout still apply
when many commands queue behind a slow Check(). Tracked separately for a
follow-up that parallelizes Check() per job.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

- README: correct admission docs to mention both errors that the
  blocking enqueue path can return (503 stopping, 500 invalid).
- manager.go dispatchInvocation: log unexpected scheduler errors via
  Warningf and respond 500 with err detail instead of swallowing them
  as a generic 503. Default branch is unreachable today but should
  surface loudly if a new error variant is added.
- SD wait_decision_test: add regression case asserting that a second
  config sent while WaitingForDecision is true blocks until the
  matching enable arrives, and then proceeds. Guards against a
  regression that would let configs interleave or accidentally clear
  the gate.

Two changes addressing the same root cause: dyncfg enable/disable
commands wedging in 'accepted' when CANCEL arrives before the function
reaches jobmgr.

1. defaultQueueSize: 64 -> 1 (kept as constant, not removed).
   Every downstream stage is single-threaded today (1 worker, 1 jobmgr
   loop, serial Check()), so the only thing a 64-deep scheduler buys us
   is wedge surface: more queued requests = more cancellation candidates.
   With queueSize=1 the stdin reader back-pressures earlier through the
   OS pipe instead of admitting work it cannot process. If real
   downstream concurrency is added later, raise this again.

2. requestCancellation: for stateQueued, ignore the cancel entirely
   (no cancelRequested flag, no ctx cancellation, no fallback timer, no
   tombstone). Reason: dyncfg commands carry side-effects that must
   reach jobmgr; without those side-effects the wait gate stays
   'accepted' forever (since the wait-decision timeout was removed).
   Setting cancelRequested would make startInvocation skip the handler;
   a fallback-timer'd tryFinalize would tombstone + remove from
   invState before the worker pulls it. Either path still wedges. The
   queued function now runs to completion as if no cancel happened;
   netdata already considers the transaction done (it 504'd before
   sending CANCEL), so the eventual terminal response just produces a
   benign "transaction not found" log on the netdata side.

Behavior for stateRunning / stateAwaitingResult is unchanged.

Updated the corresponding flow-scenario test to assert the new
contract: queued cancel does not emit 499, function still runs, normal
terminal response goes through.