Detailed Description of the Problem

An haproxy worker terminates on SIGILL from the assertion in sc_notify():

FATAL: bug condition "tick_is_expired(task->expire, now_ms)" matched at src/stconn.c:1003

The crashing code is the task-requeue path at the end of sc_notify() (src/stconn.c, identical in 3.4.0 and current master):

else {
        /* Update expiration date for the task and requeue it if not already expired.
         * Only I/O timeouts are evaluated. The stream is responsible of others.
         */
        if (!tick_is_expired(task->expire, now_ms)) {
                task->expire = tick_first(task->expire, sc_ep_rcv_ex(sc));
                task->expire = tick_first(task->expire, sc_ep_snd_ex(sc));
                task->expire = tick_first(task->expire, sc_ep_rcv_ex(sco));
                task->expire = tick_first(task->expire, sc_ep_snd_ex(sco));

                BUG_ON(tick_is_expired(task->expire, now_ms));
                task_queue(task);
        }
}

The guard only tests the incoming task->expire. It then folds in the four endpoint I/O expirations via tick_first() (which keeps the nearest tick). If any of those endpoint expirations is already in the past, task->expire becomes expired and the BUG_ON() on the next line trips.

For an applet, sc_ep_rcv_ex(sc) is sc->sedesc->lra + sc->ioto (see include/haproxy/sc_strm.h). With a short ioto and a stream connector whose last-read-activity (lra) is already older than ioto when this else-branch runs, sc_ep_rcv_ex(sc) is in the past, so the merged task->expire is in the past and the assertion fires.

We hit this in production on a Lua TCP applet (<LUA_TCP>) with a 100 ms I/O timeout. From the core dump:

sc_ep_rcv_ex(sc) = lra(4294951749) + ioto(100) = 4294951849, exactly the post-merge task->expire — already 101 ms in the past. The read I/O timeout had lapsed before sc_notify()'s else-branch executed.

This is a BUG_ON() the author explicitly intended to remove before release. Added in 08d7169f4 ("MINOR: stconn: Don't queue stream task in past in sc_notify()", 2023-11-06) with:

/* WARNING: Don't forget to remove this BUG_ON before 2.9.0 */
BUG_ON(tick_is_expired(task->expire, now_ms));

and the commit message states: "a BUG_ON() is added to be sure it never happens. I guess a good idea could be to remove it or change it to BUG_ON_HOT() for the final release." It was never removed/downgraded and is still present (and fatal under the default DEBUG_STRICT=1) in 3.4.0 and master. 78021ee9e ("BUG/MEDIUM: stconn: Don't update stream expiration date if already expired", 2023-11-09) added the surrounding if (!tick_is_expired(...)) guard, but that guard tests the value before the endpoint-timer merges, so it does not prevent the merged value from being expired.

Expected Behavior

A stream connector whose I/O timeout has already lapsed by the time sc_notify() evaluates it should not abort the process. The task should either be woken immediately to process the (already due) timeout, or be queued at the current tick — but not crash.

Steps to Reproduce the Behavior

We do not yet have a reduced standalone reproducer; it is intermittent (3 of 16 nodes, once each shortly after a worker start under production load). The conditions are:

1. A service backed by an applet (in our case a Lua tcp applet, <LUA_TCP>) so the task runs through task_process_applet() → sc_applet_process() → sc_notify().
2. A short I/O timeout on the connector (sc->ioto = 100 ms here).
3. A moment where sc_notify() takes the else-branch (task not woken for I/O) while the connector's sc_ep_rcv_ex() (lra + ioto) is already in the past — i.e. the read timeout lapsed between wakeups.

Do you have any idea what may have caused this?

Yes — see the detailed description. The if (!tick_is_expired(task->expire, now_ms)) guard validates only the pre-merge task->expire. The subsequent tick_first() merges of the four endpoint I/O expirations (sc_ep_rcv_ex/snd_ex for sc and sco) can pull task->expire into the past (an applet's lra + ioto is easily already expired with a short ioto), after which the BUG_ON() — which the author flagged for removal before 2.9.0 — fires.

Do you have an idea how to solve the issue?

Re-test after the merges and handle an already-expired result by waking the task rather than queueing it in the past, and drop/downgrade the temporary assertion per the original TODO:

if (!tick_is_expired(task->expire, now_ms)) {
        task->expire = tick_first(task->expire, sc_ep_rcv_ex(sc));
        task->expire = tick_first(task->expire, sc_ep_snd_ex(sc));
        task->expire = tick_first(task->expire, sc_ep_rcv_ex(sco));
        task->expire = tick_first(task->expire, sc_ep_snd_ex(sco));

        if (tick_is_expired(task->expire, now_ms))
                task_wakeup(task, TASK_WOKEN_TIMER);
        else
                task_queue(task);
}

At minimum, the BUG_ON() should be removed or changed to BUG_ON_HOT() as the author originally noted, since a task computed to expire in the past is a recoverable condition, not a fatal invariant violation.

What is your configuration?

global
    lua-load /etc/haproxy/rate_limit.lua

frontend lua_svc
    mode tcp
    timeout client 100ms
    timeout server 100ms
    tcp-request content use-service lua.<applet_name>

# Sanitized representative shape. The relevant factors are: mode tcp, a Lua tcp
# applet service, and ~100 ms I/O timeouts. Production front end is a consul-proxy.

Output of haproxy -vv

HAProxy version 3.4.1-1+cf1 2026/07/06 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2031.
Known bugs: http://www.haproxy.org/bugs/bugs-3.4.1.html
Running on: Linux 6.18.38-redacted x86_64

Build options :
  TARGET  = linux-glibc
  CC      = x86_64-linux-gnu-gcc
  CFLAGS  = -O2 -g -fwrapv -fvect-cost-model=very-cheap -g -O2 -Werror=implicit-function-declaration -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   =

Feature list : ... +BACKTRACE ... +LUA ... +OPENSSL ... +PROMEX ... +THREAD ...
Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=180).
Built with SSL library version : BoringSSL   (FIPS mode : yes)
Built with Lua version : Lua 5.4.7
Built with gcc compiler version 14.2.0

Last Outputs and Backtraces

Program terminated with signal SIGILL, Illegal instruction.
FATAL: bug condition "tick_is_expired(task->expire, now_ms)" matched at src/stconn.c:1003

#0  sc_notify (sc=sc@entry=0x71ce15c2f6a0)              at src/stconn.c:1003
#1  sc_applet_process (sc=sc@entry=0x71ce15c2f6a0)      at src/stconn.c:2170
#2  task_process_applet (t=0x71ce15ee6480, context=0x71ce185f4380, state=<optimized out>)
                                                        at src/applet.c:993
#3  run_tasks_from_lists (budgets=<optimized out>)      at include/haproxy/tinfo.h:127
#4  process_runnable_tasks ()                           at src/task.c:913
#5  run_poll_loop ()                                    at src/haproxy.c:3013
#6  run_thread_poll_loop (data=<optimized out>)         at src/haproxy.c:3244
#7  main (argc=<optimized out>, argv=...)               at src/haproxy.c:4020

Applet identity (frame #2): ((struct appctx*)context)->applet->name = "<LUA_TCP>"

Additional Information

• This is a BoringSSL-FIPS build. Our only local patch is SSL-compatibility (touches ssl_* / openssl-compat.h / jwe.c only) and does not touch src/stconn.c. The affected function and the assertion are verbatim upstream code (3.4.0 and current master cfe708d04), so this is not FIPS-specific — it is reachable on any standard DEBUG_STRICT=1 build.
• Behaviour is self-healing: the worker is restarted and the condition does not recur immediately; it occurred once per affected node shortly after start.
• History of the assertion: introduced 08d7169f4 (2023-11-06), surrounding guard added 78021ee9e (2023-11-09), both by Christopher Faulet.
• Confirmed present and unchanged in master (HEAD cfe708d04, 2026-07-08).