Skip to content

gh-151613: Fix remote debugging frame cache ABA - #151614

Merged
pablogsal merged 2 commits into
python:mainfrom
pablogsal:gh-151613-remote-debugging-frame-cache-aba
Jun 27, 2026
Merged

gh-151613: Fix remote debugging frame cache ABA#151614
pablogsal merged 2 commits into
python:mainfrom
pablogsal:gh-151613-remote-debugging-frame-cache-aba

Conversation

@pablogsal

@pablogsal pablogsal commented Jun 17, 2026

Copy link
Copy Markdown
Member

Fixes #151613.

The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack.

This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches.

@pablogsal
pablogsal force-pushed the gh-151613-remote-debugging-frame-cache-aba branch from 000eedb to 9f447d8 Compare June 17, 2026 21:25
@pablogsal
pablogsal marked this pull request as ready for review June 17, 2026 21:31
@pablogsal
pablogsal force-pushed the gh-151613-remote-debugging-frame-cache-aba branch 2 times, most recently from 418a947 to a548b24 Compare June 17, 2026 21:46
Fixes python#151613.

The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack.

This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches.
@pablogsal
pablogsal force-pushed the gh-151613-remote-debugging-frame-cache-aba branch from a548b24 to fc9fafd Compare June 17, 2026 22:06
@pablogsal
pablogsal requested a review from AA-Turner as a code owner June 17, 2026 22:06
Comment thread Include/internal/pycore_interpframe.h
Comment thread Lib/test/test_external_inspection.py Outdated
@pablogsal

Copy link
Copy Markdown
Member Author

Discussed this a bit with @maurycy offline.

I’m going to keep this PR as the small ABA/cache-anchor fix. The sequence here is tied to last_profiled_frame: it is there to tell “same _PyInterpreterFrame address, but not the same anchored frame anymore”.

The epoch idea where we bump on any pop while profiling is active is useful to explore, but it changes the meaning of the counter and makes the cache more conservative. I think that should be a follow-up with the Oracle data and perf/cache-hit numbers.

I’m also dropping the new test from Lib/test. It’s a good witness, but it depends on timing and address reuse, so it fits better in the Oracle/stress harness than as a buildbot pass/fail test.

@pablogsal
pablogsal enabled auto-merge (squash) June 27, 2026 16:29
@pablogsal
pablogsal merged commit 8cda6ae into python:main Jun 27, 2026
54 checks passed
@pablogsal
pablogsal deleted the gh-151613-remote-debugging-frame-cache-aba branch June 27, 2026 16:56
@pablogsal pablogsal added the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jun 27, 2026
@miss-islington-app

Copy link
Copy Markdown

Thanks @pablogsal for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

@miss-islington-app

Copy link
Copy Markdown

Sorry, @pablogsal, I could not cleanly backport this to 3.15 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 8cda6ae2f1f86f2d26c29586ffc9687b410abfcf 3.15

pablogsal added a commit to pablogsal/cpython that referenced this pull request Jun 27, 2026
The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack.

This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches.

(cherry picked from commit 8cda6ae)
pablogsal added a commit that referenced this pull request Jun 27, 2026
gh-151613: Fix remote debugging frame cache ABA (#151614)

The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack.

This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches.

(cherry picked from commit 8cda6ae)
@serhiy-storchaka

Copy link
Copy Markdown
Member

Please don't forget about backports.

@maurycy

maurycy commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

@serhiy-storchaka #152448 is a manual backport by @pablogsal, I cannot remove the label

@serhiy-storchaka serhiy-storchaka removed the needs backport to 3.15 pre-release feature fixes, bugs and security fixes label Jul 17, 2026
vlad-scherbich added a commit to DataDog/dd-trace-py that referenced this pull request Aug 26, 2026
…x wheels

The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds
in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the
3.15.0b1 interpreter in our build images.

Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish
compiled cp315 wheels, and it builds them against a 3.15 newer than ours.
CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1
(python/cpython#151614), so importing that wheel's compiled modules on the b1
interpreter our images ship crashes the process. setup.py imports Cython.Build
at module level, which is why the build dies before compiling anything.

Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so
constraining 3.15 below 3.3 makes the resolver fall back to it. It has no
CPython ABI to mismatch. Verified in the failing job's own image
(manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained
resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults;
constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes
normally. cp39 and cp314 still resolve cython==3.3.0.

This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the
dedicated cp315 jobs, so package.yml returns to main's shared build matrix and
cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI
wheels, which prof-correctness could not safely load: it pins
prof-python-3.15 to python:3.15.0b1.

The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the
DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the
image and the prof-correctness pin together.
vlad-scherbich added a commit to DataDog/dd-trace-py that referenced this pull request Aug 26, 2026
…x wheels

The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds
in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the
3.15.0b1 interpreter in our build images.

Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish
compiled cp315 wheels, and it builds them against a 3.15 newer than ours.
CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1
(python/cpython#151614), so importing that wheel's compiled modules on the b1
interpreter our images ship crashes the process. setup.py imports Cython.Build
at module level, which is why the build dies before compiling anything.

Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so
constraining 3.15 below 3.3 makes the resolver fall back to it. It has no
CPython ABI to mismatch. Verified in the failing job's own image
(manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained
resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults;
constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes
normally. cp39 and cp314 still resolve cython==3.3.0.

This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the
dedicated cp315 jobs, so package.yml returns to main's shared build matrix and
cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI
wheels, which prof-correctness could not safely load: it pins
prof-python-3.15 to python:3.15.0b1.

The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the
DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the
image and the prof-correctness pin together.
vlad-scherbich added a commit to DataDog/dd-trace-py that referenced this pull request Aug 26, 2026
python/cpython#151614 added last_profiled_frame_seq in 3.15.0b4, not rc1
(pystate.h at b1/b2/b3 has no occurrence, b4 and rc1 have one), so say so
rather than letting rc1 read as the version that introduced it. The wheel is
still described as built against the rc1 layout, and the exit condition stays
at >= 3.15.0rc1 deliberately.

Also "no compiled cp315 wheel" -> "no cp315-tagged wheel": Cython 3.2.9 does
ship compiled abi3 wheels, just none matching manylinux2014 x86_64/aarch64.
vlad-scherbich added a commit to DataDog/dd-trace-py that referenced this pull request Aug 28, 2026
…x wheels

The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds
in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the
3.15.0b1 interpreter in our build images.

Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish
compiled cp315 wheels, and it builds them against a 3.15 newer than ours.
CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1
(python/cpython#151614), so importing that wheel's compiled modules on the b1
interpreter our images ship crashes the process. setup.py imports Cython.Build
at module level, which is why the build dies before compiling anything.

Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so
constraining 3.15 below 3.3 makes the resolver fall back to it. It has no
CPython ABI to mismatch. Verified in the failing job's own image
(manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained
resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults;
constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes
normally. cp39 and cp314 still resolve cython==3.3.0.

This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the
dedicated cp315 jobs, so package.yml returns to main's shared build matrix and
cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI
wheels, which prof-correctness could not safely load: it pins
prof-python-3.15 to python:3.15.0b1.

The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the
DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the
image and the prof-correctness pin together.
vlad-scherbich added a commit to DataDog/dd-trace-py that referenced this pull request Aug 28, 2026
python/cpython#151614 added last_profiled_frame_seq in 3.15.0b4, not rc1
(pystate.h at b1/b2/b3 has no occurrence, b4 and rc1 have one), so say so
rather than letting rc1 read as the version that introduced it. The wheel is
still described as built against the rc1 layout, and the exit condition stays
at >= 3.15.0rc1 deliberately.

Also "no compiled cp315 wheel" -> "no cp315-tagged wheel": Cython 3.2.9 does
ship compiled abi3 wheels, just none matching manylinux2014 x86_64/aarch64.
vlad-scherbich added a commit to DataDog/dd-trace-py that referenced this pull request Aug 28, 2026
…x wheels

The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds
in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the
3.15.0b1 interpreter in our build images.

Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish
compiled cp315 wheels, and it builds them against a 3.15 newer than ours.
CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1
(python/cpython#151614), so importing that wheel's compiled modules on the b1
interpreter our images ship crashes the process. setup.py imports Cython.Build
at module level, which is why the build dies before compiling anything.

Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so
constraining 3.15 below 3.3 makes the resolver fall back to it. It has no
CPython ABI to mismatch. Verified in the failing job's own image
(manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained
resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults;
constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes
normally. cp39 and cp314 still resolve cython==3.3.0.

This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the
dedicated cp315 jobs, so package.yml returns to main's shared build matrix and
cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI
wheels, which prof-correctness could not safely load: it pins
prof-python-3.15 to python:3.15.0b1.

The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the
DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the
image and the prof-correctness pin together.
vlad-scherbich added a commit to DataDog/dd-trace-py that referenced this pull request Aug 28, 2026
python/cpython#151614 added last_profiled_frame_seq in 3.15.0b4, not rc1
(pystate.h at b1/b2/b3 has no occurrence, b4 and rc1 have one), so say so
rather than letting rc1 read as the version that introduced it. The wheel is
still described as built against the rc1 layout, and the exit condition stays
at >= 3.15.0rc1 deliberately.

Also "no compiled cp315 wheel" -> "no cp315-tagged wheel": Cython 3.2.9 does
ship compiled abi3 wheels, just none matching manylinux2014 x86_64/aarch64.
vlad-scherbich added a commit to DataDog/dd-trace-py that referenced this pull request Aug 28, 2026
…x wheels

The cp315 manylinux2014 wheel build started dying with SIGSEGV a few seconds
in, with no compiler output. The cause is a Cython wheel ABI mismatch, not the
3.15.0b1 interpreter in our build images.

Cython 3.3.0 (2026-08-22 05:17Z) is the first Cython release to publish
compiled cp315 wheels, and it builds them against a 3.15 newer than ours.
CPython moved members into PyThreadState between 3.15.0b1 and 3.15.0rc1
(python/cpython#151614), so importing that wheel's compiled modules on the b1
interpreter our images ship crashes the process. setup.py imports Cython.Build
at module level, which is why the build dies before compiling anything.

Cython publishes a pure-Python cython-*-py3-none-any.whl for every release, so
constraining 3.15 below 3.3 makes the resolver fall back to it. It has no
CPython ABI to mismatch. Verified in the failing job's own image
(manylinux2014_x86_64 v113741238-d2b8243, cp315 = 3.15.0b1): unconstrained
resolves cython==3.3.0 from the cp315 manylinux2014 wheel and segfaults;
constrained resolves cython==3.2.9 from the pure-Python wheel and cythonizes
normally. cp39 and cp314 still resolve cython==3.3.0.

This replaces the in-job CPython rebuild (ensure-cp315-rc1.sh) and the
dedicated cp315 jobs, so package.yml returns to main's shared build matrix and
cp315 gets warm-cache build times again. The rebuild also produced rc1-ABI
wheels, which prof-correctness could not safely load: it pins
prof-python-3.15 to python:3.15.0b1.

The constraint is a stopgap keyed to the b1/rc1 skew. Drop it when the
DataDog/images mirror ships a cp315 interpreter >= 3.15.0rc1, bumping the
image and the prof-correctness pin together.
vlad-scherbich added a commit to DataDog/dd-trace-py that referenced this pull request Aug 28, 2026
python/cpython#151614 added last_profiled_frame_seq in 3.15.0b4, not rc1
(pystate.h at b1/b2/b3 has no occurrence, b4 and rc1 have one), so say so
rather than letting rc1 read as the version that introduced it. The wheel is
still described as built against the rc1 layout, and the exit condition stays
at >= 3.15.0rc1 deliberately.

Also "no compiled cp315 wheel" -> "no cp315-tagged wheel": Cython 3.2.9 does
ship compiled abi3 wheels, just none matching manylinux2014 x86_64/aarch64.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_remote_debugging frame cache can reuse stale frame anchors

3 participants