gh-151613: Fix remote debugging frame cache ABA - #151614
Conversation
000eedb to
9f447d8
Compare
418a947 to
a548b24
Compare
Fixes python#151613. The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack. This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches.
a548b24 to
fc9fafd
Compare
|
Discussed this a bit with @maurycy offline. I’m going to keep this PR as the small ABA/cache-anchor fix. The sequence here is tied to The epoch idea where we bump on any pop while profiling is active is useful to explore, but it changes the meaning of the counter and makes the cache more conservative. I think that should be a follow-up with the Oracle data and perf/cache-hit numbers. I’m also dropping the new test from |
|
Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15. |
|
Sorry, @pablogsal, I could not cleanly backport this to |
The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack. This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches. (cherry picked from commit 8cda6ae)
gh-151613: Fix remote debugging frame cache ABA (#151614) The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack. This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches. (cherry picked from commit 8cda6ae)
|
Please don't forget about backports. |
|
@serhiy-storchaka #152448 is a manual backport by @pablogsal, I cannot remove the label |
…x wheels The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the 3.15.0b1 interpreter in our build images. Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish compiled cp315 wheels, and it builds them against a 3.15 newer than ours. CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1 (python/cpython#151614), so importing that wheel's compiled modules on the b1 interpreter our images ship crashes the process. setup.py imports Cython.Build at module level, which is why the build dies before compiling anything. Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so constraining 3.15 below 3.3 makes the resolver fall back to it. It has no CPython ABI to mismatch. Verified in the failing job's own image (manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults; constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes normally. cp39 and cp314 still resolve cython==3.3.0. This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the dedicated cp315 jobs, so package.yml returns to main's shared build matrix and cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI wheels, which prof-correctness could not safely load: it pins prof-python-3.15 to python:3.15.0b1. The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the image and the prof-correctness pin together.
…x wheels The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the 3.15.0b1 interpreter in our build images. Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish compiled cp315 wheels, and it builds them against a 3.15 newer than ours. CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1 (python/cpython#151614), so importing that wheel's compiled modules on the b1 interpreter our images ship crashes the process. setup.py imports Cython.Build at module level, which is why the build dies before compiling anything. Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so constraining 3.15 below 3.3 makes the resolver fall back to it. It has no CPython ABI to mismatch. Verified in the failing job's own image (manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults; constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes normally. cp39 and cp314 still resolve cython==3.3.0. This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the dedicated cp315 jobs, so package.yml returns to main's shared build matrix and cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI wheels, which prof-correctness could not safely load: it pins prof-python-3.15 to python:3.15.0b1. The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the image and the prof-correctness pin together.
python/cpython#151614 added last_profiled_frame_seq in 3.15.0b4, not rc1 (pystate.h at b1/b2/b3 has no occurrence, b4 and rc1 have one), so say so rather than letting rc1 read as the version that introduced it. The wheel is still described as built against the rc1 layout, and the exit condition stays at >= 3.15.0rc1 deliberately. Also "no compiled cp315 wheel" -> "no cp315-tagged wheel": Cython 3.2.9 does ship compiled abi3 wheels, just none matching manylinux2014 x86_64/aarch64.
…x wheels The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the 3.15.0b1 interpreter in our build images. Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish compiled cp315 wheels, and it builds them against a 3.15 newer than ours. CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1 (python/cpython#151614), so importing that wheel's compiled modules on the b1 interpreter our images ship crashes the process. setup.py imports Cython.Build at module level, which is why the build dies before compiling anything. Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so constraining 3.15 below 3.3 makes the resolver fall back to it. It has no CPython ABI to mismatch. Verified in the failing job's own image (manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults; constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes normally. cp39 and cp314 still resolve cython==3.3.0. This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the dedicated cp315 jobs, so package.yml returns to main's shared build matrix and cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI wheels, which prof-correctness could not safely load: it pins prof-python-3.15 to python:3.15.0b1. The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the image and the prof-correctness pin together.
python/cpython#151614 added last_profiled_frame_seq in 3.15.0b4, not rc1 (pystate.h at b1/b2/b3 has no occurrence, b4 and rc1 have one), so say so rather than letting rc1 read as the version that introduced it. The wheel is still described as built against the rc1 layout, and the exit condition stays at >= 3.15.0rc1 deliberately. Also "no compiled cp315 wheel" -> "no cp315-tagged wheel": Cython 3.2.9 does ship compiled abi3 wheels, just none matching manylinux2014 x86_64/aarch64.
…x wheels The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the 3.15.0b1 interpreter in our build images. Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish compiled cp315 wheels, and it builds them against a 3.15 newer than ours. CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1 (python/cpython#151614), so importing that wheel's compiled modules on the b1 interpreter our images ship crashes the process. setup.py imports Cython.Build at module level, which is why the build dies before compiling anything. Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so constraining 3.15 below 3.3 makes the resolver fall back to it. It has no CPython ABI to mismatch. Verified in the failing job's own image (manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults; constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes normally. cp39 and cp314 still resolve cython==3.3.0. This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the dedicated cp315 jobs, so package.yml returns to main's shared build matrix and cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI wheels, which prof-correctness could not safely load: it pins prof-python-3.15 to python:3.15.0b1. The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the image and the prof-correctness pin together.
python/cpython#151614 added last_profiled_frame_seq in 3.15.0b4, not rc1 (pystate.h at b1/b2/b3 has no occurrence, b4 and rc1 have one), so say so rather than letting rc1 read as the version that introduced it. The wheel is still described as built against the rc1 layout, and the exit condition stays at >= 3.15.0rc1 deliberately. Also "no compiled cp315 wheel" -> "no cp315-tagged wheel": Cython 3.2.9 does ship compiled abi3 wheels, just none matching manylinux2014 x86_64/aarch64.
…x wheels The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the 3.15.0b1 interpreter in our build images. Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish compiled cp315 wheels, and it builds them against a 3.15 newer than ours. CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1 (python/cpython#151614), so importing that wheel's compiled modules on the b1 interpreter our images ship crashes the process. setup.py imports Cython.Build at module level, which is why the build dies before compiling anything. Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so constraining 3.15 below 3.3 makes the resolver fall back to it. It has no CPython ABI to mismatch. Verified in the failing job's own image (manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults; constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes normally. cp39 and cp314 still resolve cython==3.3.0. This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the dedicated cp315 jobs, so package.yml returns to main's shared build matrix and cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI wheels, which prof-correctness could not safely load: it pins prof-python-3.15 to python:3.15.0b1. The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the image and the prof-correctness pin together.
python/cpython#151614 added last_profiled_frame_seq in 3.15.0b4, not rc1 (pystate.h at b1/b2/b3 has no occurrence, b4 and rc1 have one), so say so rather than letting rc1 read as the version that introduced it. The wheel is still described as built against the rc1 layout, and the exit condition stays at >= 3.15.0rc1 deliberately. Also "no compiled cp315 wheel" -> "no cp315-tagged wheel": Cython 3.2.9 does ship compiled abi3 wheels, just none matching manylinux2014 x86_64/aarch64.
Fixes #151613.
The remote debugging frame cache previously used only the
last_profiled_frameaddress as its cache anchor. If a frame returned and a later frame reused the same_PyInterpreterFrameaddress, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack.This adds a
last_profiled_frame_seqcounter next tolast_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches.