Fix member table copy-paste, exception clobbering, missing Py_VISIT#360
Merged
etrepum merged 1 commit intoApr 7, 2026
Merged
Conversation
- encoder_members: "encoding" mapped to offsetof(encoder) instead of offsetof(encoding). Copy-paste from line 181. obj.encoding returned the encoder callable instead of the encoding value. - encoder_new: PyLong_AsLong(int_as_string_bitcount) returns -1 on overflow without checking PyErr_Occurred(). The -1 failed the > 0 check, falling to PyErr_Format(TypeError) which clobbered the real OverflowError. - encoder_traverse: add missing Py_VISIT(s->skipkeys_bool) to match encoder_clear's Py_CLEAR(s->skipkeys_bool). Closes simplejson#354 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
etrepum
approved these changes
Apr 6, 2026
auto-merge was automatically disabled
April 6, 2026 17:41
Merge commits are not allowed on this repository
etrepum
pushed a commit
that referenced
this pull request
Apr 11, 2026
…ct, SKIP_WHITESPACE, field X-macros, n format Five mechanical cleanups to the C extension, none of which change behavior. Together they remove ~190 lines of duplication and close several classes of recurring bug. encoder_markers_push / encoder_markers_pop (previously duplicated in 3 places): The circular-reference marker pattern — PyLong_FromVoidPtr(obj), PyDict_Contains, PyDict_SetItem on push, and PyDict_DelItem + Py_DECREF on pop — appeared verbatim in encoder_listencode_obj, encoder_listencode_dict, and encoder_listencode_list. Three recent bug fixes (#358, #360, aa9182d) patched individual sites; factoring into two helpers collapses ~60 lines, and any future fix lives in one place. The NULL-sentinel convention on ident lets callers invoke markers_pop unconditionally on the happy path. encoder_listencode_default extraction: The inner else { ... } of encoder_listencode_obj (RawJSON + iterable fallback + markers-tracked defaultfn recursion) lived inline with nested `break` into an outer do { } while(0) and a stray indentation level from an unbraced scope. Extract it verbatim into its own function that returns 0/-1 directly, so the main dispatch loop is a clean chain of else-if arms with no `break` inside the final arm. SKIP_WHITESPACE() macro in _speedups_scan.h: `while (idx <= end_idx && IS_WHITESPACE(JSON_SCAN_READ(idx))) idx++;` appeared 8 times across _parse_object and _parse_array. Collapse to a macro defined alongside JSON_SCAN_FN / JSON_SCAN_CONCAT, #undef'd at the bottom of the template so the multi-include pattern stays hygienic. PyArg_ParseTuple "n" format code replaces _convertPyInt_AsSsize_t / _convertPyInt_FromSsize_t: The custom O& converter predates broad "n" (Py_ssize_t) support in PyArg_ParseTuple and Py_BuildValue. Both have supported "n" since Python 2.5 — the simplejson floor — so we can drop the two wrappers and use "n" directly in py_scanstring, scanner_call, encoder_call, and raise_errmsg. Saves an indirect function call per parse on three hot entry points. JSON_SCANNER_OBJECT_FIELDS / JSON_ENCODER_OBJECT_FIELDS X-macros: scanner_traverse + scanner_clear and encoder_traverse + encoder_clear all listed the same fields 2x — an easy place to forget a field when adding one (exactly the bug fixed in c23e6d9). Collapse to an X-macro field list adjacent to each struct definition, used with JSON_VISIT_FIELD / JSON_CLEAR_FIELD local expansions. Adding a new PyObject* field now needs one line in the X-macro, not two in each of four different functions. Verification: - Strict CFLAGS build on CPython 3.11: -Wall -Wextra -Wshadow -Wstrict-prototypes -Wdeclaration-after-statement -Werror, clean - Default CFLAGS build on CPython 3.14.0rc2 free-threaded: clean - Full _cibw_runner suite on both (354 tests, C + pure-Python paths): 354/354 pass - Targeted correctness tests on 3.14t: for_json / _asdict / default / iterable_as_array / RawJSON / circular detection on all three encoder sites (dict, list, default) - 16-thread x 5000-iter stress on a shared JSONDecoder and JSONEncoder with the GIL disabled: no mismatches, no races https://claude.ai/code/session_011EfS4WKeHCX3xPsmHuvCnz
github-merge-queue Bot
pushed a commit
that referenced
this pull request
Apr 12, 2026
* Modernize scanner/encoder dict ops: SetDefault for memo intern, GetItemRef for key_memo Two targeted cleanups to the scanner/encoder hot paths, motivated by reviewing the code that landed in #367 and comparing against CPython's _json.c and PR #344. No behavior change; just fewer dict lookups and cleaner use of the modern strong-reference dict APIs available on Python 3.13+. Scanner (_parse_object memo intern): The GetItemWithError -> Py_INCREF / PyDict_SetItem dance did two hashtable probes for every fresh key. Collapse it to a single PyDict_SetDefault (or PyDict_SetDefaultRef on 3.13+), which atomically gets-or-sets in one pass. Factored into json_memo_intern_key so the _unicode and _str template instantiations share one implementation, and the 3.13+ fast path is isolated in one place. The `memokey` temporary is gone, the loop body drops 13 lines to 6, and unique-key JSON decoding touches the memo dict half as often. Encoder (key_memo cache lookup in encoder_listencode_dict): Replace the GetItemWithError + manual Py_INCREF + PyErr_Occurred check with a call to a new json_PyDict_GetItemRef helper. On 3.13+ this forwards to PyDict_GetItemRef, which atomically returns a strong reference and eliminates the borrowed-reference window that is technically racy under free threading even under the coarse self critical section. On older Pythons the helper falls back to the legacy idiom. The caller becomes a single rc-based branch, and the Py_CLEAR(kstr) is no longer duplicated across three arms. Both changes compile cleanly under -Wall -Wextra -Wshadow -Wstrict-prototypes -Wdeclaration-after-statement -Werror on CPython 3.11, and under the default CFLAGS on CPython 3.14.0rc2 free-threaded. Full _cibw_runner suite (354 tests, C + pure-Python passes) passes on both. 16-thread x 5000-iter stress test on a shared JSONDecoder / JSONEncoder passes with the GIL disabled. Explicitly not changed: - Py_BEGIN_CRITICAL_SECTION(self) in scanner_call and encoder_call. The scanner needs it because PyDict_Clear(s->memo) at end-of-call would race with concurrent scan_once calls if we switched to a per-dict lock; the encoder uses it defensively but c_make_encoder is called fresh per JSONEncoder.iterencode() call in the normal API flow, so the lock is uncontended in practice. Fine-grained container locks (CPython-style, see PR #344 discussion) would only help the unusual case of an explicitly shared encoder across threads, and the win does not justify the refactor. https://claude.ai/code/session_011EfS4WKeHCX3xPsmHuvCnz * Deduplicate scanner/encoder hot paths: markers helpers, default extract, SKIP_WHITESPACE, field X-macros, n format Five mechanical cleanups to the C extension, none of which change behavior. Together they remove ~190 lines of duplication and close several classes of recurring bug. encoder_markers_push / encoder_markers_pop (previously duplicated in 3 places): The circular-reference marker pattern — PyLong_FromVoidPtr(obj), PyDict_Contains, PyDict_SetItem on push, and PyDict_DelItem + Py_DECREF on pop — appeared verbatim in encoder_listencode_obj, encoder_listencode_dict, and encoder_listencode_list. Three recent bug fixes (#358, #360, aa9182d) patched individual sites; factoring into two helpers collapses ~60 lines, and any future fix lives in one place. The NULL-sentinel convention on ident lets callers invoke markers_pop unconditionally on the happy path. encoder_listencode_default extraction: The inner else { ... } of encoder_listencode_obj (RawJSON + iterable fallback + markers-tracked defaultfn recursion) lived inline with nested `break` into an outer do { } while(0) and a stray indentation level from an unbraced scope. Extract it verbatim into its own function that returns 0/-1 directly, so the main dispatch loop is a clean chain of else-if arms with no `break` inside the final arm. SKIP_WHITESPACE() macro in _speedups_scan.h: `while (idx <= end_idx && IS_WHITESPACE(JSON_SCAN_READ(idx))) idx++;` appeared 8 times across _parse_object and _parse_array. Collapse to a macro defined alongside JSON_SCAN_FN / JSON_SCAN_CONCAT, #undef'd at the bottom of the template so the multi-include pattern stays hygienic. PyArg_ParseTuple "n" format code replaces _convertPyInt_AsSsize_t / _convertPyInt_FromSsize_t: The custom O& converter predates broad "n" (Py_ssize_t) support in PyArg_ParseTuple and Py_BuildValue. Both have supported "n" since Python 2.5 — the simplejson floor — so we can drop the two wrappers and use "n" directly in py_scanstring, scanner_call, encoder_call, and raise_errmsg. Saves an indirect function call per parse on three hot entry points. JSON_SCANNER_OBJECT_FIELDS / JSON_ENCODER_OBJECT_FIELDS X-macros: scanner_traverse + scanner_clear and encoder_traverse + encoder_clear all listed the same fields 2x — an easy place to forget a field when adding one (exactly the bug fixed in c23e6d9). Collapse to an X-macro field list adjacent to each struct definition, used with JSON_VISIT_FIELD / JSON_CLEAR_FIELD local expansions. Adding a new PyObject* field now needs one line in the X-macro, not two in each of four different functions. Verification: - Strict CFLAGS build on CPython 3.11: -Wall -Wextra -Wshadow -Wstrict-prototypes -Wdeclaration-after-statement -Werror, clean - Default CFLAGS build on CPython 3.14.0rc2 free-threaded: clean - Full _cibw_runner suite on both (354 tests, C + pure-Python paths): 354/354 pass - Targeted correctness tests on 3.14t: for_json / _asdict / default / iterable_as_array / RawJSON / circular detection on all three encoder sites (dict, list, default) - 16-thread x 5000-iter stress on a shared JSONDecoder and JSONEncoder with the GIL disabled: no mismatches, no races https://claude.ai/code/session_011EfS4WKeHCX3xPsmHuvCnz * Encoder cleanup: all-string-keys dict sort fast path, T_OBJECT -> Py_T_OBJECT_EX Two independent improvements bundled together because they touch adjacent code. #5 — encoder_dict_iteritems fast path for all-string keys: Sorted dict encoding with sort_keys=True (or a custom item_sort_key) used to go through a double-iteration loop: PyDict_Items produced a list, then the code walked it with PyIter_Next, type-checked each key, and PyList_Append'd a rebuilt list to sort. For the overwhelmingly common case of string-keyed JSON objects this was all wasted work — every tuple was kept verbatim and the "slow" rebuild list was just a duplicate of the items list. Add a fast path: if every key in the items list is already a JSON- compatible string (PyUnicode on all versions, plus PyString on Python 2), sort `items` in place via the shared encoder_sort_items_inplace helper and return iter(items). No per-item tuple reallocation, no list alloc, no stringify branch in the hot loop. On any non-string key the pre-scan bails out and falls through to the existing stringify-and-rebuild path, so the slow path is preserved exactly as before. Factored the list.sort() call into encoder_sort_items_inplace and the "is this a JSON string key" test into is_json_string_key so the two paths share one source of truth. Measured on CPython 3.14t free-threaded, 200-entry string-keyed dict with 3-element list values: sort_keys=True is now 0.204 ms/op vs 0.197 ms/op for the unsorted path — ~4% overhead, essentially just the cost of sorting itself. Previously the double-walk and list rebuild added substantial constant-factor overhead on top. #6 — T_OBJECT -> Py_T_OBJECT_EX on all member descriptors: T_OBJECT is deprecated in Python 3.12+ in favor of the new public spelling Py_T_OBJECT_EX. The semantic difference is that T_OBJECT returns Py_None when the underlying slot is NULL, while Py_T_OBJECT_EX raises AttributeError. Keep the Python-visible behavior unchanged by: 1. Defining Py_T_OBJECT_EX to T_OBJECT_EX on pre-3.12 (both available via <structmember.h>, identical semantics), so the modern spelling compiles on the full 2.5+ version range simplejson supports. 2. Switching encoder_new to store Py_None rather than NULL when encoding=None on Python 3, so the .encoding attribute still returns None (as it did under T_OBJECT) rather than raising AttributeError under Py_T_OBJECT_EX. 3. Updating the two bytes-handling sentinel checks (encoder_stringify_key and encoder_listencode_obj) from `s->encoding != NULL` to `s->encoding != Py_None` so the internal "is encoding configured" test matches the new representation. All 20 members across scanner_members and encoder_members updated in one pass. Verification: - Strict CFLAGS on CPython 3.11: -Wall -Wextra -Wshadow -Wstrict-prototypes -Wdeclaration-after-statement -Werror, clean - Default CFLAGS on CPython 3.14.0rc2 free-threaded: clean - Full _cibw_runner suite (354 tests, C + pure-Python) on both: OK - Targeted tests for encoder_dict_iteritems paths: regular dict / OrderedDict / dict subclass / empty dict / sort_keys=True with all string keys (fast path) / mixed string-and-int keys (slow path) / int keys / float keys / skipkeys+non-string / custom item_sort_key / unicode keys — all pass - encoding=None on Py3 round-trip + bytes-key rejection with encoding=None: behavior preserved - 16-thread x 5000-iter stress on shared JSONEncoder with sort_keys=True under free threading: no mismatches https://claude.ai/code/session_011EfS4WKeHCX3xPsmHuvCnz --------- Co-authored-by: Claude <noreply@anthropic.com>
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
Apr 20, 2026
Version 4.0.1 released 2026-04-18 * Skip uploading Pyodide/wasm wheels to PyPI, which rejects them with "unsupported platform tag 'pyodide_2024_0_wasm32'". The wheels are still built in CI and preserved as workflow artifacts. simplejson/simplejson#375 Version 4.0.0 released 2026-04-18 * simplejson 4 requires Python 2.7 or Python 3.8+. Older Python versions (2.5, 2.6, 3.0-3.7) are no longer supported. pip will not install simplejson 4 on unsupported versions. * The C extension now uses heap types and per-module state instead of static types and global state. This is required for free-threading support and sub-interpreter isolation. The Python-level API is unchanged. * Full support for Python 3.13+ free-threading (PEP 703). The C extension is now safe to use with the GIL disabled (python3.14t): - Converted all static types to heap types with per-module state - Added per-object critical sections to scanner and encoder - Added free-threading-safe dict operations for Python 3.13+ - Unified per-module state management and templated parser simplejson/simplejson#363 simplejson/simplejson#364 simplejson/simplejson#365 simplejson/simplejson#367 simplejson/simplejson#369 * Numerous C extension memory safety fixes: - Fix use-after-free and leak in encoder ident handling - Fix NULL dereferences on OOM in module init and static string init - Fix reference leaks in dict encoder (skipkeys item, variable shadowing) - Fix member table copy-paste, exception clobbering, missing Py_VISIT - Fix error-as-truthy bugs in maybe_quote_bigint and is_raw_json - Fix iterable_as_array swallowing MemoryError and KeyboardInterrupt - Fix for_json and _asdict swallowing MemoryError, KeyboardInterrupt, and other non-AttributeError exceptions raised by user __getattr__ simplejson/simplejson#355 simplejson/simplejson#356 simplejson/simplejson#357 simplejson/simplejson#358 simplejson/simplejson#359 simplejson/simplejson#360 simplejson/simplejson#373 * C/Python parity fixes: - Fix C scanstring off-by-one bounds checks that caused truncated or boundary \uXXXX escapes to raise "Invalid \\uXXXX escape sequence" instead of "Unterminated string", and report error position at the 'u' instead of the leading backslash. The C and Python decoders now agree on exception class, message, and position across all tested edge cases. - Align the Python encoder's dispatch order with the C encoder for objects that define _asdict(). Previously a list/tuple/dict subclass with an _asdict() method encoded as its container type under the Python encoder and as the _asdict() return value under the C encoder; both now check _asdict() before list/tuple/dict. for_json() continues to outrank _asdict() in both. - Fix C scanstring raising a plain ValueError ("end is out of bounds") instead of JSONDecodeError for out-of-range end indices. User code with `except JSONDecodeError:` now catches both the C and pure-Python paths consistently. simplejson/simplejson#372 * C extension performance and correctness improvements: - Add PyDict_Next fast path for unsorted exact-dict encoding, avoiding intermediate items list and N tuple allocations - Add indexed fast path for exact list/tuple encoding, avoiding iterator allocation and per-item PyIter_Next overhead - Use PyUnicodeWriter as JSON_Accu backend on Python 3.14+, eliminating intermediate string objects and ''.join calls - Fix integer overflow in ascii_escape output_size calculation that could cause buffer overwrite on pathologically large strings - Fix list encoder separator counter overflow (int to Py_ssize_t) - Dead code cleanup (unreachable NULL checks, do-while wrappers) simplejson/simplejson#370 * Added Python 3.14 support and updated to cibuildwheel 3.2.1. CI now tests free-threaded (3.14t) and debug builds with -Werror, refcount leak detection, and GIL-disabled mode. simplejson/simplejson#343 * Added a ThreadSanitizer (TSan) stress test CI job. Builds a TSan-instrumented free-threaded CPython (cached between runs) and runs a concurrent stress test script against the C extension to catch data races under free-threading. simplejson/simplejson#373 * Replace deprecated license classifiers with SPDX license expression simplejson/simplejson#347 * Documented RawJSON usage with examples and caveats simplejson/simplejson#346 * Added pyproject.toml for PEP 517 build support. setup.py is retained for Python 2.7 wheel builds and backwards compatibility. * Migrated build_ext import from distutils to setuptools in setup.py. The distutils.errors imports are kept since setuptools vendors distutils on Python 3.12+ where stdlib distutils was removed. * CI now tests PEP 517 builds (pyproject.toml) alongside the existing setup.py-based builds. * Added Pyodide (wasm32) wheel builds with C speedups via cibuildwheel. Previously Pyodide users fell back to the pure-Python wheel; now they get the compiled C extension cross-compiled to WebAssembly. Thread and subprocess tests are skipped on Emscripten where those APIs are unavailable. * Test suite now fails (instead of skipping) when C speedups are missing during cibuildwheel runs, catching broken extension builds early. * New ``array_hook`` parameter for ``loads()``, ``load()``, and ``JSONDecoder``. Called with each decoded JSON array (as a list), its return value replaces the list. Analogous to ``object_hook`` for dicts. Works with both the Python decoder and C scanner. (Matches CPython 3.15 json module.) * Trailing comma detection: the decoder now raises ``JSONDecodeError`` with "Illegal trailing comma before end of object/array" for inputs like ``[1,]`` and ``{"a": 1,}`` instead of generic error messages. Both the Python decoder and C scanner are updated. (Matches CPython 3.13+ json module.) * ``frozendict`` encoding support: when ``frozendict`` is available (CPython 3.15+ PEP 814), it is encoded as a JSON object just like ``dict``. No effect on older Python versions. * Serialization errors now include ``add_note()`` context on Python 3.11+ (PEP 678), annotating exceptions with the path to the error, e.g. "when serializing list item 1" / "when serializing dict item 'key'". Only applies to the Python encoder. * New C fast path for ``encode_basestring`` (``ensure_ascii=False``). Previously the non-ASCII string encoder fell back to pure Python; now it has a C implementation matching the existing ``encode_basestring_ascii`` fast path. simplejson/simplejson#207 * The Python decoder now rejects non-ASCII digits (e.g. fullwidth ``\uff10``) in JSON numbers, matching the C scanner behavior. The ``NUMBER_RE`` regex was changed from ``\d`` to ``[0-9]``. * Removed dead single-phase init code for Python 3.3/3.4 from the C extension (these versions are no longer supported).
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
May 13, 2026
Version 4.0.1 released 2026-04-18 * Skip uploading Pyodide/wasm wheels to PyPI, which rejects them with "unsupported platform tag 'pyodide_2024_0_wasm32'". The wheels are still built in CI and preserved as workflow artifacts. simplejson/simplejson#375 Version 4.0.0 released 2026-04-18 * simplejson 4 requires Python 2.7 or Python 3.8+. Older Python versions (2.5, 2.6, 3.0-3.7) are no longer supported. pip will not install simplejson 4 on unsupported versions. * The C extension now uses heap types and per-module state instead of static types and global state. This is required for free-threading support and sub-interpreter isolation. The Python-level API is unchanged. * Full support for Python 3.13+ free-threading (PEP 703). The C extension is now safe to use with the GIL disabled (python3.14t): - Converted all static types to heap types with per-module state - Added per-object critical sections to scanner and encoder - Added free-threading-safe dict operations for Python 3.13+ - Unified per-module state management and templated parser simplejson/simplejson#363 simplejson/simplejson#364 simplejson/simplejson#365 simplejson/simplejson#367 simplejson/simplejson#369 * Numerous C extension memory safety fixes: - Fix use-after-free and leak in encoder ident handling - Fix NULL dereferences on OOM in module init and static string init - Fix reference leaks in dict encoder (skipkeys item, variable shadowing) - Fix member table copy-paste, exception clobbering, missing Py_VISIT - Fix error-as-truthy bugs in maybe_quote_bigint and is_raw_json - Fix iterable_as_array swallowing MemoryError and KeyboardInterrupt - Fix for_json and _asdict swallowing MemoryError, KeyboardInterrupt, and other non-AttributeError exceptions raised by user __getattr__ simplejson/simplejson#355 simplejson/simplejson#356 simplejson/simplejson#357 simplejson/simplejson#358 simplejson/simplejson#359 simplejson/simplejson#360 simplejson/simplejson#373 * C/Python parity fixes: - Fix C scanstring off-by-one bounds checks that caused truncated or boundary \uXXXX escapes to raise "Invalid \\uXXXX escape sequence" instead of "Unterminated string", and report error position at the 'u' instead of the leading backslash. The C and Python decoders now agree on exception class, message, and position across all tested edge cases. - Align the Python encoder's dispatch order with the C encoder for objects that define _asdict(). Previously a list/tuple/dict subclass with an _asdict() method encoded as its container type under the Python encoder and as the _asdict() return value under the C encoder; both now check _asdict() before list/tuple/dict. for_json() continues to outrank _asdict() in both. - Fix C scanstring raising a plain ValueError ("end is out of bounds") instead of JSONDecodeError for out-of-range end indices. User code with `except JSONDecodeError:` now catches both the C and pure-Python paths consistently. simplejson/simplejson#372 * C extension performance and correctness improvements: - Add PyDict_Next fast path for unsorted exact-dict encoding, avoiding intermediate items list and N tuple allocations - Add indexed fast path for exact list/tuple encoding, avoiding iterator allocation and per-item PyIter_Next overhead - Use PyUnicodeWriter as JSON_Accu backend on Python 3.14+, eliminating intermediate string objects and ''.join calls - Fix integer overflow in ascii_escape output_size calculation that could cause buffer overwrite on pathologically large strings - Fix list encoder separator counter overflow (int to Py_ssize_t) - Dead code cleanup (unreachable NULL checks, do-while wrappers) simplejson/simplejson#370 * Added Python 3.14 support and updated to cibuildwheel 3.2.1. CI now tests free-threaded (3.14t) and debug builds with -Werror, refcount leak detection, and GIL-disabled mode. simplejson/simplejson#343 * Added a ThreadSanitizer (TSan) stress test CI job. Builds a TSan-instrumented free-threaded CPython (cached between runs) and runs a concurrent stress test script against the C extension to catch data races under free-threading. simplejson/simplejson#373 * Replace deprecated license classifiers with SPDX license expression simplejson/simplejson#347 * Documented RawJSON usage with examples and caveats simplejson/simplejson#346 * Added pyproject.toml for PEP 517 build support. setup.py is retained for Python 2.7 wheel builds and backwards compatibility. * Migrated build_ext import from distutils to setuptools in setup.py. The distutils.errors imports are kept since setuptools vendors distutils on Python 3.12+ where stdlib distutils was removed. * CI now tests PEP 517 builds (pyproject.toml) alongside the existing setup.py-based builds. * Added Pyodide (wasm32) wheel builds with C speedups via cibuildwheel. Previously Pyodide users fell back to the pure-Python wheel; now they get the compiled C extension cross-compiled to WebAssembly. Thread and subprocess tests are skipped on Emscripten where those APIs are unavailable. * Test suite now fails (instead of skipping) when C speedups are missing during cibuildwheel runs, catching broken extension builds early. * New ``array_hook`` parameter for ``loads()``, ``load()``, and ``JSONDecoder``. Called with each decoded JSON array (as a list), its return value replaces the list. Analogous to ``object_hook`` for dicts. Works with both the Python decoder and C scanner. (Matches CPython 3.15 json module.) * Trailing comma detection: the decoder now raises ``JSONDecodeError`` with "Illegal trailing comma before end of object/array" for inputs like ``[1,]`` and ``{"a": 1,}`` instead of generic error messages. Both the Python decoder and C scanner are updated. (Matches CPython 3.13+ json module.) * ``frozendict`` encoding support: when ``frozendict`` is available (CPython 3.15+ PEP 814), it is encoded as a JSON object just like ``dict``. No effect on older Python versions. * Serialization errors now include ``add_note()`` context on Python 3.11+ (PEP 678), annotating exceptions with the path to the error, e.g. "when serializing list item 1" / "when serializing dict item 'key'". Only applies to the Python encoder. * New C fast path for ``encode_basestring`` (``ensure_ascii=False``). Previously the non-ASCII string encoder fell back to pure Python; now it has a C implementation matching the existing ``encode_basestring_ascii`` fast path. simplejson/simplejson#207 * The Python decoder now rejects non-ASCII digits (e.g. fullwidth ``\uff10``) in JSON numbers, matching the C scanner behavior. The ``NUMBER_RE`` regex was changed from ``\d`` to ``[0-9]``. * Removed dead single-phase init code for Python 3.3/3.4 from the C extension (these versions are no longer supported).
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
May 21, 2026
Version 4.0.1 released 2026-04-18 * Skip uploading Pyodide/wasm wheels to PyPI, which rejects them with "unsupported platform tag 'pyodide_2024_0_wasm32'". The wheels are still built in CI and preserved as workflow artifacts. simplejson/simplejson#375 Version 4.0.0 released 2026-04-18 * simplejson 4 requires Python 2.7 or Python 3.8+. Older Python versions (2.5, 2.6, 3.0-3.7) are no longer supported. pip will not install simplejson 4 on unsupported versions. * The C extension now uses heap types and per-module state instead of static types and global state. This is required for free-threading support and sub-interpreter isolation. The Python-level API is unchanged. * Full support for Python 3.13+ free-threading (PEP 703). The C extension is now safe to use with the GIL disabled (python3.14t): - Converted all static types to heap types with per-module state - Added per-object critical sections to scanner and encoder - Added free-threading-safe dict operations for Python 3.13+ - Unified per-module state management and templated parser simplejson/simplejson#363 simplejson/simplejson#364 simplejson/simplejson#365 simplejson/simplejson#367 simplejson/simplejson#369 * Numerous C extension memory safety fixes: - Fix use-after-free and leak in encoder ident handling - Fix NULL dereferences on OOM in module init and static string init - Fix reference leaks in dict encoder (skipkeys item, variable shadowing) - Fix member table copy-paste, exception clobbering, missing Py_VISIT - Fix error-as-truthy bugs in maybe_quote_bigint and is_raw_json - Fix iterable_as_array swallowing MemoryError and KeyboardInterrupt - Fix for_json and _asdict swallowing MemoryError, KeyboardInterrupt, and other non-AttributeError exceptions raised by user __getattr__ simplejson/simplejson#355 simplejson/simplejson#356 simplejson/simplejson#357 simplejson/simplejson#358 simplejson/simplejson#359 simplejson/simplejson#360 simplejson/simplejson#373 * C/Python parity fixes: - Fix C scanstring off-by-one bounds checks that caused truncated or boundary \uXXXX escapes to raise "Invalid \\uXXXX escape sequence" instead of "Unterminated string", and report error position at the 'u' instead of the leading backslash. The C and Python decoders now agree on exception class, message, and position across all tested edge cases. - Align the Python encoder's dispatch order with the C encoder for objects that define _asdict(). Previously a list/tuple/dict subclass with an _asdict() method encoded as its container type under the Python encoder and as the _asdict() return value under the C encoder; both now check _asdict() before list/tuple/dict. for_json() continues to outrank _asdict() in both. - Fix C scanstring raising a plain ValueError ("end is out of bounds") instead of JSONDecodeError for out-of-range end indices. User code with `except JSONDecodeError:` now catches both the C and pure-Python paths consistently. simplejson/simplejson#372 * C extension performance and correctness improvements: - Add PyDict_Next fast path for unsorted exact-dict encoding, avoiding intermediate items list and N tuple allocations - Add indexed fast path for exact list/tuple encoding, avoiding iterator allocation and per-item PyIter_Next overhead - Use PyUnicodeWriter as JSON_Accu backend on Python 3.14+, eliminating intermediate string objects and ''.join calls - Fix integer overflow in ascii_escape output_size calculation that could cause buffer overwrite on pathologically large strings - Fix list encoder separator counter overflow (int to Py_ssize_t) - Dead code cleanup (unreachable NULL checks, do-while wrappers) simplejson/simplejson#370 * Added Python 3.14 support and updated to cibuildwheel 3.2.1. CI now tests free-threaded (3.14t) and debug builds with -Werror, refcount leak detection, and GIL-disabled mode. simplejson/simplejson#343 * Added a ThreadSanitizer (TSan) stress test CI job. Builds a TSan-instrumented free-threaded CPython (cached between runs) and runs a concurrent stress test script against the C extension to catch data races under free-threading. simplejson/simplejson#373 * Replace deprecated license classifiers with SPDX license expression simplejson/simplejson#347 * Documented RawJSON usage with examples and caveats simplejson/simplejson#346 * Added pyproject.toml for PEP 517 build support. setup.py is retained for Python 2.7 wheel builds and backwards compatibility. * Migrated build_ext import from distutils to setuptools in setup.py. The distutils.errors imports are kept since setuptools vendors distutils on Python 3.12+ where stdlib distutils was removed. * CI now tests PEP 517 builds (pyproject.toml) alongside the existing setup.py-based builds. * Added Pyodide (wasm32) wheel builds with C speedups via cibuildwheel. Previously Pyodide users fell back to the pure-Python wheel; now they get the compiled C extension cross-compiled to WebAssembly. Thread and subprocess tests are skipped on Emscripten where those APIs are unavailable. * Test suite now fails (instead of skipping) when C speedups are missing during cibuildwheel runs, catching broken extension builds early. * New ``array_hook`` parameter for ``loads()``, ``load()``, and ``JSONDecoder``. Called with each decoded JSON array (as a list), its return value replaces the list. Analogous to ``object_hook`` for dicts. Works with both the Python decoder and C scanner. (Matches CPython 3.15 json module.) * Trailing comma detection: the decoder now raises ``JSONDecodeError`` with "Illegal trailing comma before end of object/array" for inputs like ``[1,]`` and ``{"a": 1,}`` instead of generic error messages. Both the Python decoder and C scanner are updated. (Matches CPython 3.13+ json module.) * ``frozendict`` encoding support: when ``frozendict`` is available (CPython 3.15+ PEP 814), it is encoded as a JSON object just like ``dict``. No effect on older Python versions. * Serialization errors now include ``add_note()`` context on Python 3.11+ (PEP 678), annotating exceptions with the path to the error, e.g. "when serializing list item 1" / "when serializing dict item 'key'". Only applies to the Python encoder. * New C fast path for ``encode_basestring`` (``ensure_ascii=False``). Previously the non-ASCII string encoder fell back to pure Python; now it has a C implementation matching the existing ``encode_basestring_ascii`` fast path. simplejson/simplejson#207 * The Python decoder now rejects non-ASCII digits (e.g. fullwidth ``\uff10``) in JSON numbers, matching the C scanner behavior. The ``NUMBER_RE`` regex was changed from ``\d`` to ``[0-9]``. * Removed dead single-phase init code for Python 3.3/3.4 from the C extension (these versions are no longer supported).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #354 (part of #348).
Changes
encoder_memberscopy-paste (Finding 6):"encoding"usedoffsetof(PyEncoderObject, encoder)instead ofoffsetof(PyEncoderObject, encoding). One character fix.PyLong_AsLongexception clobbering (Finding 12):int_as_string_bitcountoverflow returned -1 withoutPyErr_Occurred()check, clobberingOverflowErrorwithTypeError.encoder_traversemissingPy_VISIT(Finding 5): AddedPy_VISIT(s->skipkeys_bool)to matchencoder_clear.Found using cext-review-toolkit.