Skip to content

3.3: Segfault because of LUA applet #3244

@idl0r

Description

@idl0r

Detailed Description of the Problem

I started to roll out 3.3.1 yesterday (from 3.2.10 prior) and got two crashes over night, while it was running fine on 3.2. for quite some time. So I suspect a bug in 3.3.

Expected Behavior

No crash

Steps to Reproduce the Behavior

N/A

Do you have any idea what may have caused this?

Something LUA related and perhaps due to a reload that happened at the same time basically.
The LUA code basically parses data from a stick-table for rate calculation. It reads from stick-table and returns it.

-- Summieren von Sticktableeinträgen aller haproxy-Instanzen mit Hostnamensuffix

local function http_req_rate(txn, tablename)
    -- Der Frontendname muss innerhalb der Konfiguration in die Variable req.fe_name geschrieben werden, z.B. via:
    -- http-request set-var(req.fe_name) hdr(X-fe_name)
    local fe_name = txn:get_var("req.fe_name")
    local sticktable = core.proxies[tablename].stktable:dump({})
    local rate = 0
    -- Datenformat der gedumpten sticktable: { ["172.17.0.1"] = { ["gpt"] = 0,["http_req_cnt"] = 3,["http_req_rate"] = 3,} ,}
    for k,v in pairs(sticktable) do
        -- Vergleich des Frontendnamen ohne 'n095xxx|'-Prefix
        -- Achtung: LUA Stringindizes beginnen bei 1!
        if string.sub(k, 9) == fe_name then
            rate = rate + v["http_req_rate"]
        end
    end

    return rate
end

core.register_fetches("http_req_rate", http_req_rate);

Do you have an idea how to solve the issue?

No response

What is your configuration?

tune.lua.bool-sample-conversion normal
lua-load-per-thread /etc/haproxy/enodia-backend-counter.lua

Output of haproxy -vv

HAProxy version 3.3.1-9c24c11 2025/12/19 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-3.3.1.html
Running on: Linux 6.1.0-42-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.159-1 (2025-12-30) x86_64
Build options : 
  TARGET  = linux-glibc
  CC      = cc
  CFLAGS  = -O2 -g -march=x86-64-v3 -mtune=znver1 -fwrapv -fvect-cost-model=very-cheap
  OPTIONS = USE_LIBCRYPT=1 USE_OPENSSL_AWSLC=1 USE_LUA=1 USE_ZLIB= USE_SLZ=1 USE_NS= USE_PROMEX=1 USE_PCRE= USE_PCRE_JIT= USE_PCRE2=1 USE_PCRE2_JIT=
  DEBUG   = -DDEBUG_GLITCHES=1

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ECH -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +KTLS -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER -NS -OBSOLETE_LINKER +OPENSSL +OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB +ACME

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=64).
Built with SSL library version : OpenSSL 1.1.1 (compatible; AWS-LC 1.66.2)
Running on SSL library version : AWS-LC 1.66.2
SSL library supports TLS extensions : yes
SSL library supports SNI : yes
SSL library FIPS mode : no
SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 12.2.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
       spop : mode=SPOP  side=BE     mux=SPOP  flags=HOL_RISK|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

Last Outputs and Backtraces

Jan 12 22:38:08 n255027 systemd[1]: Reloaded haproxy.service - HAProxy Load Balancer.
Jan 12 22:38:08 n255027 haproxy[39149]: [NOTICE]   (39149) : haproxy version is 3.3.1-9c24c11
Jan 12 22:38:08 n255027 haproxy[39149]: [NOTICE]   (39149) : path to executable is /usr/sbin/haproxy
Jan 12 22:38:08 n255027 haproxy[39149]: [WARNING]  (39149) : Former worker (132907) exited with code 0 (Exit)
Jan 12 22:38:16 n255027 haproxy[169678]: [ALERT]    (169678) : proxy 'genlisten_25941-aa_schulung_app_static_delivery' has no server available!
Jan 12 22:38:16 n255027 haproxy[169678]: [ALERT]    (169678) : proxy 'genlisten_34041-bbk_prod_www_headless_server' has no server available!
Jan 12 22:38:16 n255027 haproxy[169678]: [ALERT]    (169678) : proxy 'genlisten_34241-bbk_stage_www_headless_server' has no server available!
Jan 12 22:38:17 n255027 haproxy[169678]: [ALERT]    (169678) : proxy 'genlisten_aa_integration_www' has no server available!
Jan 12 23:12:34 n255027 haproxy[169678]: [ALERT]    (169678) : proxy 'genlisten_17440-henkel_stage_www' has no server available!
Jan 12 23:12:44 n255027 haproxy[169678]: [ALERT]    (169678) : proxy 'genlisten_62440-wtv_staging_www' has no server available!
Jan 12 23:12:51 n255027 haproxy[169678]: [ALERT]    (169678) : proxy 'genlisten_24240-bmzin_staging_www' has no server available!
Jan 12 23:15:16 n255027 haproxy[169678]:   call trace(13):
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x55963d59d06d <02 00 00 e8 a3 8b f7 ff]: main+0x25a5bd > ha_backtrace_to_stderr
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x55963d59e54d <44 89 e6 e8 43 e9 ff ff]: stats_fill_sv_line+0x2fd/0x127a > main+0x25a3e0
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x55963d3c4022 <fe ff ff e8 2e a2 1d 00]: hlua_process_task+0x12612 > stats_fill_sv_line
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x55963d3c8f39 <fe ff ff e8 57 ac ff ff]: hlua_process_task+0x17529 > hlua_process_task+0x12180
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x55963d535e24 <40 48 8b 43 58 ff 50 18]: task_process_applet+0x344/0xa84
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x55963d5dc2f9 <84 6a 02 00 00 41 ff d2]: run_tasks_from_lists+0x3d9/0x9fd
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x55963d5dcd09 <4e 30 01 e8 17 f2 ff ff]: process_runnable_tasks+0x3e9/0xa3f > run_tasks_from_lists
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x55963d52b5f5 <01 00 00 e8 2b 13 0b 00]: run_poll_loop+0x135/0x5a1 > process_runnable_tasks
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x55963d52bcd1 <00 00 00 e8 ef f7 ff ff]: run_thread_poll_loop+0x261/0x567 > run_poll_loop
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x7f0637f331f5 <00 00 ff 90 38 06 00 00]: libc:+0x891f5
Jan 12 23:15:16 n255027 haproxy[169678]:   | 0x7f0637fb38dc <83 e4 f0 4c 89 c7 ff d2]: libc:+0x1098dc
Jan 12 23:15:16 n255027 haproxy[169678]: Hint: when reporting this bug to developers, please check if a core file was
Jan 12 23:15:16 n255027 haproxy[169678]:       produced, open it with 'gdb', issue 'bt' to produce a backtrace for the
Jan 12 23:15:16 n255027 haproxy[169678]:       current thread only, then join it with the bug report.
Jan 12 23:15:16 n255027 haproxy[39149]: [ALERT]    (39149) : Current worker (169678) exited with code 132 (Illegal instruction)
Jan 12 23:15:16 n255027 haproxy[39149]: [WARNING]  (39149) : A worker process unexpectedly died and this can only be explained by a bug in haproxy or its dependencies.
Jan 12 23:15:16 n255027 haproxy[39149]: Please check that you are running an up to date and maintained version of haproxy and open a bug report.
Jan 12 23:15:16 n255027 haproxy[39149]: HAProxy version 3.3.1-9c24c11 2025/12/19 - https://haproxy.org/
Jan 12 23:15:16 n255027 haproxy[39149]: Status: stable branch - will stop receiving fixes around Q1 2027.
Jan 12 23:15:16 n255027 haproxy[39149]: Known bugs: http://www.haproxy.org/bugs/bugs-3.3.1.html
Jan 12 23:15:16 n255027 haproxy[39149]: Running on: Linux 6.1.0-42-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.159-1 (2025-12-30) x86_64
Jan 12 23:15:16 n255027 haproxy[39149]: [ALERT]    (39149) : exit-on-failure: killing every processes with SIGTERM
Jan 12 23:15:16 n255027 haproxy[39149]: [WARNING]  (39149) : All workers exited. Exiting... (132)

Additional Information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.0This issue affects the HAProxy 3.0 stable branch.3.2The issue affects the HAProxy 3.2 stable branch.3.3The issue affects the HAProxy 3.3 stable branch.status: fixedThis issue is a now-fixed bug.type: bugThis issue describes a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions