Skip to content

Fix member table copy-paste, exception clobbering, missing Py_VISIT#360

Merged
etrepum merged 1 commit into
simplejson:masterfrom
devdanzin:fix/minor-member-exception-traverse
Apr 7, 2026
Merged

Fix member table copy-paste, exception clobbering, missing Py_VISIT#360
etrepum merged 1 commit into
simplejson:masterfrom
devdanzin:fix/minor-member-exception-traverse

Conversation

@devdanzin

Copy link
Copy Markdown
Contributor

Fixes #354 (part of #348).

Changes

  1. encoder_members copy-paste (Finding 6): "encoding" used offsetof(PyEncoderObject, encoder) instead of offsetof(PyEncoderObject, encoding). One character fix.

  2. PyLong_AsLong exception clobbering (Finding 12): int_as_string_bitcount overflow returned -1 without PyErr_Occurred() check, clobbering OverflowError with TypeError.

  3. encoder_traverse missing Py_VISIT (Finding 5): Added Py_VISIT(s->skipkeys_bool) to match encoder_clear.

Found using cext-review-toolkit.

- encoder_members: "encoding" mapped to offsetof(encoder) instead of
  offsetof(encoding). Copy-paste from line 181. obj.encoding returned
  the encoder callable instead of the encoding value.

- encoder_new: PyLong_AsLong(int_as_string_bitcount) returns -1 on
  overflow without checking PyErr_Occurred(). The -1 failed the > 0
  check, falling to PyErr_Format(TypeError) which clobbered the real
  OverflowError.

- encoder_traverse: add missing Py_VISIT(s->skipkeys_bool) to match
  encoder_clear's Py_CLEAR(s->skipkeys_bool).

Closes simplejson#354

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@etrepum etrepum enabled auto-merge April 6, 2026 16:17
@etrepum etrepum disabled auto-merge April 6, 2026 17:32
@etrepum etrepum enabled auto-merge April 6, 2026 17:32
auto-merge was automatically disabled April 6, 2026 17:41

Merge commits are not allowed on this repository

@etrepum etrepum added this pull request to the merge queue Apr 6, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks Apr 6, 2026
@etrepum etrepum added this pull request to the merge queue Apr 7, 2026
Merged via the queue into simplejson:master with commit c23e6d9 Apr 7, 2026
9 checks passed
etrepum pushed a commit that referenced this pull request Apr 11, 2026
…ct, SKIP_WHITESPACE, field X-macros, n format

Five mechanical cleanups to the C extension, none of which change
behavior. Together they remove ~190 lines of duplication and close
several classes of recurring bug.

encoder_markers_push / encoder_markers_pop (previously duplicated in 3
places):
  The circular-reference marker pattern — PyLong_FromVoidPtr(obj),
  PyDict_Contains, PyDict_SetItem on push, and PyDict_DelItem +
  Py_DECREF on pop — appeared verbatim in encoder_listencode_obj,
  encoder_listencode_dict, and encoder_listencode_list. Three recent
  bug fixes (#358, #360, aa9182d) patched individual sites; factoring
  into two helpers collapses ~60 lines, and any future fix lives in
  one place. The NULL-sentinel convention on ident lets callers invoke
  markers_pop unconditionally on the happy path.

encoder_listencode_default extraction:
  The inner else { ... } of encoder_listencode_obj (RawJSON + iterable
  fallback + markers-tracked defaultfn recursion) lived inline with
  nested `break` into an outer do { } while(0) and a stray indentation
  level from an unbraced scope. Extract it verbatim into its own
  function that returns 0/-1 directly, so the main dispatch loop is a
  clean chain of else-if arms with no `break` inside the final arm.

SKIP_WHITESPACE() macro in _speedups_scan.h:
  `while (idx <= end_idx && IS_WHITESPACE(JSON_SCAN_READ(idx))) idx++;`
  appeared 8 times across _parse_object and _parse_array. Collapse to
  a macro defined alongside JSON_SCAN_FN / JSON_SCAN_CONCAT, #undef'd
  at the bottom of the template so the multi-include pattern stays
  hygienic.

PyArg_ParseTuple "n" format code replaces _convertPyInt_AsSsize_t /
_convertPyInt_FromSsize_t:
  The custom O& converter predates broad "n" (Py_ssize_t) support in
  PyArg_ParseTuple and Py_BuildValue. Both have supported "n" since
  Python 2.5 — the simplejson floor — so we can drop the two wrappers
  and use "n" directly in py_scanstring, scanner_call, encoder_call,
  and raise_errmsg. Saves an indirect function call per parse on three
  hot entry points.

JSON_SCANNER_OBJECT_FIELDS / JSON_ENCODER_OBJECT_FIELDS X-macros:
  scanner_traverse + scanner_clear and encoder_traverse +
  encoder_clear all listed the same fields 2x — an easy place to
  forget a field when adding one (exactly the bug fixed in c23e6d9).
  Collapse to an X-macro field list adjacent to each struct
  definition, used with JSON_VISIT_FIELD / JSON_CLEAR_FIELD local
  expansions. Adding a new PyObject* field now needs one line in the
  X-macro, not two in each of four different functions.

Verification:
- Strict CFLAGS build on CPython 3.11: -Wall -Wextra -Wshadow
  -Wstrict-prototypes -Wdeclaration-after-statement -Werror, clean
- Default CFLAGS build on CPython 3.14.0rc2 free-threaded: clean
- Full _cibw_runner suite on both (354 tests, C + pure-Python paths):
  354/354 pass
- Targeted correctness tests on 3.14t: for_json / _asdict / default /
  iterable_as_array / RawJSON / circular detection on all three
  encoder sites (dict, list, default)
- 16-thread x 5000-iter stress on a shared JSONDecoder and
  JSONEncoder with the GIL disabled: no mismatches, no races

https://claude.ai/code/session_011EfS4WKeHCX3xPsmHuvCnz
github-merge-queue Bot pushed a commit that referenced this pull request Apr 12, 2026
* Modernize scanner/encoder dict ops: SetDefault for memo intern, GetItemRef for key_memo

Two targeted cleanups to the scanner/encoder hot paths, motivated by
reviewing the code that landed in #367 and comparing against CPython's
_json.c and PR #344. No behavior change; just fewer dict lookups and
cleaner use of the modern strong-reference dict APIs available on
Python 3.13+.

Scanner (_parse_object memo intern):
  The GetItemWithError -> Py_INCREF / PyDict_SetItem dance did two
  hashtable probes for every fresh key. Collapse it to a single
  PyDict_SetDefault (or PyDict_SetDefaultRef on 3.13+), which atomically
  gets-or-sets in one pass. Factored into json_memo_intern_key so the
  _unicode and _str template instantiations share one implementation,
  and the 3.13+ fast path is isolated in one place. The `memokey`
  temporary is gone, the loop body drops 13 lines to 6, and unique-key
  JSON decoding touches the memo dict half as often.

Encoder (key_memo cache lookup in encoder_listencode_dict):
  Replace the GetItemWithError + manual Py_INCREF + PyErr_Occurred check
  with a call to a new json_PyDict_GetItemRef helper. On 3.13+ this
  forwards to PyDict_GetItemRef, which atomically returns a strong
  reference and eliminates the borrowed-reference window that is
  technically racy under free threading even under the coarse self
  critical section. On older Pythons the helper falls back to the
  legacy idiom. The caller becomes a single rc-based branch, and the
  Py_CLEAR(kstr) is no longer duplicated across three arms.

Both changes compile cleanly under -Wall -Wextra -Wshadow
-Wstrict-prototypes -Wdeclaration-after-statement -Werror on CPython
3.11, and under the default CFLAGS on CPython 3.14.0rc2 free-threaded.
Full _cibw_runner suite (354 tests, C + pure-Python passes) passes on
both. 16-thread x 5000-iter stress test on a shared JSONDecoder /
JSONEncoder passes with the GIL disabled.

Explicitly not changed:
- Py_BEGIN_CRITICAL_SECTION(self) in scanner_call and encoder_call.
  The scanner needs it because PyDict_Clear(s->memo) at end-of-call
  would race with concurrent scan_once calls if we switched to a
  per-dict lock; the encoder uses it defensively but c_make_encoder
  is called fresh per JSONEncoder.iterencode() call in the normal
  API flow, so the lock is uncontended in practice. Fine-grained
  container locks (CPython-style, see PR #344 discussion) would only
  help the unusual case of an explicitly shared encoder across
  threads, and the win does not justify the refactor.

https://claude.ai/code/session_011EfS4WKeHCX3xPsmHuvCnz

* Deduplicate scanner/encoder hot paths: markers helpers, default extract, SKIP_WHITESPACE, field X-macros, n format

Five mechanical cleanups to the C extension, none of which change
behavior. Together they remove ~190 lines of duplication and close
several classes of recurring bug.

encoder_markers_push / encoder_markers_pop (previously duplicated in 3
places):
  The circular-reference marker pattern — PyLong_FromVoidPtr(obj),
  PyDict_Contains, PyDict_SetItem on push, and PyDict_DelItem +
  Py_DECREF on pop — appeared verbatim in encoder_listencode_obj,
  encoder_listencode_dict, and encoder_listencode_list. Three recent
  bug fixes (#358, #360, aa9182d) patched individual sites; factoring
  into two helpers collapses ~60 lines, and any future fix lives in
  one place. The NULL-sentinel convention on ident lets callers invoke
  markers_pop unconditionally on the happy path.

encoder_listencode_default extraction:
  The inner else { ... } of encoder_listencode_obj (RawJSON + iterable
  fallback + markers-tracked defaultfn recursion) lived inline with
  nested `break` into an outer do { } while(0) and a stray indentation
  level from an unbraced scope. Extract it verbatim into its own
  function that returns 0/-1 directly, so the main dispatch loop is a
  clean chain of else-if arms with no `break` inside the final arm.

SKIP_WHITESPACE() macro in _speedups_scan.h:
  `while (idx <= end_idx && IS_WHITESPACE(JSON_SCAN_READ(idx))) idx++;`
  appeared 8 times across _parse_object and _parse_array. Collapse to
  a macro defined alongside JSON_SCAN_FN / JSON_SCAN_CONCAT, #undef'd
  at the bottom of the template so the multi-include pattern stays
  hygienic.

PyArg_ParseTuple "n" format code replaces _convertPyInt_AsSsize_t /
_convertPyInt_FromSsize_t:
  The custom O& converter predates broad "n" (Py_ssize_t) support in
  PyArg_ParseTuple and Py_BuildValue. Both have supported "n" since
  Python 2.5 — the simplejson floor — so we can drop the two wrappers
  and use "n" directly in py_scanstring, scanner_call, encoder_call,
  and raise_errmsg. Saves an indirect function call per parse on three
  hot entry points.

JSON_SCANNER_OBJECT_FIELDS / JSON_ENCODER_OBJECT_FIELDS X-macros:
  scanner_traverse + scanner_clear and encoder_traverse +
  encoder_clear all listed the same fields 2x — an easy place to
  forget a field when adding one (exactly the bug fixed in c23e6d9).
  Collapse to an X-macro field list adjacent to each struct
  definition, used with JSON_VISIT_FIELD / JSON_CLEAR_FIELD local
  expansions. Adding a new PyObject* field now needs one line in the
  X-macro, not two in each of four different functions.

Verification:
- Strict CFLAGS build on CPython 3.11: -Wall -Wextra -Wshadow
  -Wstrict-prototypes -Wdeclaration-after-statement -Werror, clean
- Default CFLAGS build on CPython 3.14.0rc2 free-threaded: clean
- Full _cibw_runner suite on both (354 tests, C + pure-Python paths):
  354/354 pass
- Targeted correctness tests on 3.14t: for_json / _asdict / default /
  iterable_as_array / RawJSON / circular detection on all three
  encoder sites (dict, list, default)
- 16-thread x 5000-iter stress on a shared JSONDecoder and
  JSONEncoder with the GIL disabled: no mismatches, no races

https://claude.ai/code/session_011EfS4WKeHCX3xPsmHuvCnz

* Encoder cleanup: all-string-keys dict sort fast path, T_OBJECT -> Py_T_OBJECT_EX

Two independent improvements bundled together because they touch
adjacent code.

#5 — encoder_dict_iteritems fast path for all-string keys:

  Sorted dict encoding with sort_keys=True (or a custom item_sort_key)
  used to go through a double-iteration loop: PyDict_Items produced
  a list, then the code walked it with PyIter_Next, type-checked each
  key, and PyList_Append'd a rebuilt list to sort. For the
  overwhelmingly common case of string-keyed JSON objects this was
  all wasted work — every tuple was kept verbatim and the "slow"
  rebuild list was just a duplicate of the items list.

  Add a fast path: if every key in the items list is already a JSON-
  compatible string (PyUnicode on all versions, plus PyString on
  Python 2), sort `items` in place via the shared
  encoder_sort_items_inplace helper and return iter(items). No
  per-item tuple reallocation, no list alloc, no stringify branch
  in the hot loop.

  On any non-string key the pre-scan bails out and falls through to
  the existing stringify-and-rebuild path, so the slow path is
  preserved exactly as before. Factored the list.sort() call into
  encoder_sort_items_inplace and the "is this a JSON string key"
  test into is_json_string_key so the two paths share one source of
  truth.

  Measured on CPython 3.14t free-threaded, 200-entry string-keyed
  dict with 3-element list values: sort_keys=True is now 0.204 ms/op
  vs 0.197 ms/op for the unsorted path — ~4% overhead, essentially
  just the cost of sorting itself. Previously the double-walk and
  list rebuild added substantial constant-factor overhead on top.

#6 — T_OBJECT -> Py_T_OBJECT_EX on all member descriptors:

  T_OBJECT is deprecated in Python 3.12+ in favor of the new public
  spelling Py_T_OBJECT_EX. The semantic difference is that
  T_OBJECT returns Py_None when the underlying slot is NULL, while
  Py_T_OBJECT_EX raises AttributeError. Keep the Python-visible
  behavior unchanged by:

  1. Defining Py_T_OBJECT_EX to T_OBJECT_EX on pre-3.12 (both
     available via <structmember.h>, identical semantics), so the
     modern spelling compiles on the full 2.5+ version range
     simplejson supports.
  2. Switching encoder_new to store Py_None rather than NULL when
     encoding=None on Python 3, so the .encoding attribute still
     returns None (as it did under T_OBJECT) rather than raising
     AttributeError under Py_T_OBJECT_EX.
  3. Updating the two bytes-handling sentinel checks
     (encoder_stringify_key and encoder_listencode_obj) from
     `s->encoding != NULL` to `s->encoding != Py_None` so the
     internal "is encoding configured" test matches the new
     representation.

  All 20 members across scanner_members and encoder_members updated
  in one pass.

Verification:

- Strict CFLAGS on CPython 3.11: -Wall -Wextra -Wshadow
  -Wstrict-prototypes -Wdeclaration-after-statement -Werror, clean
- Default CFLAGS on CPython 3.14.0rc2 free-threaded: clean
- Full _cibw_runner suite (354 tests, C + pure-Python) on both: OK
- Targeted tests for encoder_dict_iteritems paths: regular dict /
  OrderedDict / dict subclass / empty dict / sort_keys=True with all
  string keys (fast path) / mixed string-and-int keys (slow path) /
  int keys / float keys / skipkeys+non-string / custom item_sort_key
  / unicode keys — all pass
- encoding=None on Py3 round-trip + bytes-key rejection with
  encoding=None: behavior preserved
- 16-thread x 5000-iter stress on shared JSONEncoder with
  sort_keys=True under free threading: no mismatches

https://claude.ai/code/session_011EfS4WKeHCX3xPsmHuvCnz

---------

Co-authored-by: Claude <noreply@anthropic.com>
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Apr 20, 2026
Version 4.0.1 released 2026-04-18

* Skip uploading Pyodide/wasm wheels to PyPI, which rejects them with
  "unsupported platform tag 'pyodide_2024_0_wasm32'". The wheels are
  still built in CI and preserved as workflow artifacts.
  simplejson/simplejson#375

Version 4.0.0 released 2026-04-18

* simplejson 4 requires Python 2.7 or Python 3.8+. Older Python
  versions (2.5, 2.6, 3.0-3.7) are no longer supported. pip will
  not install simplejson 4 on unsupported versions.

* The C extension now uses heap types and per-module state instead of
  static types and global state. This is required for free-threading
  support and sub-interpreter isolation. The Python-level API is
  unchanged.

* Full support for Python 3.13+ free-threading (PEP 703). The C
  extension is now safe to use with the GIL disabled (python3.14t):
  - Converted all static types to heap types with per-module state
  - Added per-object critical sections to scanner and encoder
  - Added free-threading-safe dict operations for Python 3.13+
  - Unified per-module state management and templated parser
  simplejson/simplejson#363
  simplejson/simplejson#364
  simplejson/simplejson#365
  simplejson/simplejson#367
  simplejson/simplejson#369

* Numerous C extension memory safety fixes:
  - Fix use-after-free and leak in encoder ident handling
  - Fix NULL dereferences on OOM in module init and static string init
  - Fix reference leaks in dict encoder (skipkeys item, variable shadowing)
  - Fix member table copy-paste, exception clobbering, missing Py_VISIT
  - Fix error-as-truthy bugs in maybe_quote_bigint and is_raw_json
  - Fix iterable_as_array swallowing MemoryError and KeyboardInterrupt
  - Fix for_json and _asdict swallowing MemoryError, KeyboardInterrupt,
    and other non-AttributeError exceptions raised by user __getattr__
  simplejson/simplejson#355
  simplejson/simplejson#356
  simplejson/simplejson#357
  simplejson/simplejson#358
  simplejson/simplejson#359
  simplejson/simplejson#360
  simplejson/simplejson#373

* C/Python parity fixes:
  - Fix C scanstring off-by-one bounds checks that caused truncated
    or boundary \uXXXX escapes to raise "Invalid \\uXXXX escape
    sequence" instead of "Unterminated string", and report error
    position at the 'u' instead of the leading backslash. The C and
    Python decoders now agree on exception class, message, and
    position across all tested edge cases.
  - Align the Python encoder's dispatch order with the C encoder for
    objects that define _asdict(). Previously a list/tuple/dict
    subclass with an _asdict() method encoded as its container type
    under the Python encoder and as the _asdict() return value under
    the C encoder; both now check _asdict() before list/tuple/dict.
    for_json() continues to outrank _asdict() in both.
  - Fix C scanstring raising a plain ValueError ("end is out of
    bounds") instead of JSONDecodeError for out-of-range end indices.
    User code with `except JSONDecodeError:` now catches both the
    C and pure-Python paths consistently.
  simplejson/simplejson#372

* C extension performance and correctness improvements:
  - Add PyDict_Next fast path for unsorted exact-dict encoding,
    avoiding intermediate items list and N tuple allocations
  - Add indexed fast path for exact list/tuple encoding, avoiding
    iterator allocation and per-item PyIter_Next overhead
  - Use PyUnicodeWriter as JSON_Accu backend on Python 3.14+,
    eliminating intermediate string objects and ''.join calls
  - Fix integer overflow in ascii_escape output_size calculation
    that could cause buffer overwrite on pathologically large strings
  - Fix list encoder separator counter overflow (int to Py_ssize_t)
  - Dead code cleanup (unreachable NULL checks, do-while wrappers)
  simplejson/simplejson#370

* Added Python 3.14 support and updated to cibuildwheel 3.2.1. CI now
  tests free-threaded (3.14t) and debug builds with -Werror, refcount
  leak detection, and GIL-disabled mode.
  simplejson/simplejson#343

* Added a ThreadSanitizer (TSan) stress test CI job. Builds a
  TSan-instrumented free-threaded CPython (cached between runs) and
  runs a concurrent stress test script against the C extension to
  catch data races under free-threading.
  simplejson/simplejson#373

* Replace deprecated license classifiers with SPDX license expression
  simplejson/simplejson#347

* Documented RawJSON usage with examples and caveats
  simplejson/simplejson#346

* Added pyproject.toml for PEP 517 build support. setup.py is retained
  for Python 2.7 wheel builds and backwards compatibility.

* Migrated build_ext import from distutils to setuptools in setup.py.
  The distutils.errors imports are kept since setuptools vendors
  distutils on Python 3.12+ where stdlib distutils was removed.

* CI now tests PEP 517 builds (pyproject.toml) alongside the existing
  setup.py-based builds.

* Added Pyodide (wasm32) wheel builds with C speedups via cibuildwheel.
  Previously Pyodide users fell back to the pure-Python wheel; now they
  get the compiled C extension cross-compiled to WebAssembly. Thread
  and subprocess tests are skipped on Emscripten where those APIs are
  unavailable.

* Test suite now fails (instead of skipping) when C speedups are missing
  during cibuildwheel runs, catching broken extension builds early.

* New ``array_hook`` parameter for ``loads()``, ``load()``, and
  ``JSONDecoder``. Called with each decoded JSON array (as a list),
  its return value replaces the list. Analogous to ``object_hook``
  for dicts. Works with both the Python decoder and C scanner.
  (Matches CPython 3.15 json module.)

* Trailing comma detection: the decoder now raises ``JSONDecodeError``
  with "Illegal trailing comma before end of object/array" for inputs
  like ``[1,]`` and ``{"a": 1,}`` instead of generic error messages.
  Both the Python decoder and C scanner are updated.
  (Matches CPython 3.13+ json module.)

* ``frozendict`` encoding support: when ``frozendict`` is available
  (CPython 3.15+ PEP 814), it is encoded as a JSON object just like
  ``dict``. No effect on older Python versions.

* Serialization errors now include ``add_note()`` context on Python
  3.11+ (PEP 678), annotating exceptions with the path to the error,
  e.g. "when serializing list item 1" / "when serializing dict item
  'key'". Only applies to the Python encoder.

* New C fast path for ``encode_basestring`` (``ensure_ascii=False``).
  Previously the non-ASCII string encoder fell back to pure Python;
  now it has a C implementation matching the existing
  ``encode_basestring_ascii`` fast path.
  simplejson/simplejson#207

* The Python decoder now rejects non-ASCII digits (e.g. fullwidth
  ``\uff10``) in JSON numbers, matching the C scanner behavior.
  The ``NUMBER_RE`` regex was changed from ``\d`` to ``[0-9]``.

* Removed dead single-phase init code for Python 3.3/3.4 from the
  C extension (these versions are no longer supported).
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request May 13, 2026
Version 4.0.1 released 2026-04-18

* Skip uploading Pyodide/wasm wheels to PyPI, which rejects them with
  "unsupported platform tag 'pyodide_2024_0_wasm32'". The wheels are
  still built in CI and preserved as workflow artifacts.
  simplejson/simplejson#375

Version 4.0.0 released 2026-04-18

* simplejson 4 requires Python 2.7 or Python 3.8+. Older Python
  versions (2.5, 2.6, 3.0-3.7) are no longer supported. pip will
  not install simplejson 4 on unsupported versions.

* The C extension now uses heap types and per-module state instead of
  static types and global state. This is required for free-threading
  support and sub-interpreter isolation. The Python-level API is
  unchanged.

* Full support for Python 3.13+ free-threading (PEP 703). The C
  extension is now safe to use with the GIL disabled (python3.14t):
  - Converted all static types to heap types with per-module state
  - Added per-object critical sections to scanner and encoder
  - Added free-threading-safe dict operations for Python 3.13+
  - Unified per-module state management and templated parser
  simplejson/simplejson#363
  simplejson/simplejson#364
  simplejson/simplejson#365
  simplejson/simplejson#367
  simplejson/simplejson#369

* Numerous C extension memory safety fixes:
  - Fix use-after-free and leak in encoder ident handling
  - Fix NULL dereferences on OOM in module init and static string init
  - Fix reference leaks in dict encoder (skipkeys item, variable shadowing)
  - Fix member table copy-paste, exception clobbering, missing Py_VISIT
  - Fix error-as-truthy bugs in maybe_quote_bigint and is_raw_json
  - Fix iterable_as_array swallowing MemoryError and KeyboardInterrupt
  - Fix for_json and _asdict swallowing MemoryError, KeyboardInterrupt,
    and other non-AttributeError exceptions raised by user __getattr__
  simplejson/simplejson#355
  simplejson/simplejson#356
  simplejson/simplejson#357
  simplejson/simplejson#358
  simplejson/simplejson#359
  simplejson/simplejson#360
  simplejson/simplejson#373

* C/Python parity fixes:
  - Fix C scanstring off-by-one bounds checks that caused truncated
    or boundary \uXXXX escapes to raise "Invalid \\uXXXX escape
    sequence" instead of "Unterminated string", and report error
    position at the 'u' instead of the leading backslash. The C and
    Python decoders now agree on exception class, message, and
    position across all tested edge cases.
  - Align the Python encoder's dispatch order with the C encoder for
    objects that define _asdict(). Previously a list/tuple/dict
    subclass with an _asdict() method encoded as its container type
    under the Python encoder and as the _asdict() return value under
    the C encoder; both now check _asdict() before list/tuple/dict.
    for_json() continues to outrank _asdict() in both.
  - Fix C scanstring raising a plain ValueError ("end is out of
    bounds") instead of JSONDecodeError for out-of-range end indices.
    User code with `except JSONDecodeError:` now catches both the
    C and pure-Python paths consistently.
  simplejson/simplejson#372

* C extension performance and correctness improvements:
  - Add PyDict_Next fast path for unsorted exact-dict encoding,
    avoiding intermediate items list and N tuple allocations
  - Add indexed fast path for exact list/tuple encoding, avoiding
    iterator allocation and per-item PyIter_Next overhead
  - Use PyUnicodeWriter as JSON_Accu backend on Python 3.14+,
    eliminating intermediate string objects and ''.join calls
  - Fix integer overflow in ascii_escape output_size calculation
    that could cause buffer overwrite on pathologically large strings
  - Fix list encoder separator counter overflow (int to Py_ssize_t)
  - Dead code cleanup (unreachable NULL checks, do-while wrappers)
  simplejson/simplejson#370

* Added Python 3.14 support and updated to cibuildwheel 3.2.1. CI now
  tests free-threaded (3.14t) and debug builds with -Werror, refcount
  leak detection, and GIL-disabled mode.
  simplejson/simplejson#343

* Added a ThreadSanitizer (TSan) stress test CI job. Builds a
  TSan-instrumented free-threaded CPython (cached between runs) and
  runs a concurrent stress test script against the C extension to
  catch data races under free-threading.
  simplejson/simplejson#373

* Replace deprecated license classifiers with SPDX license expression
  simplejson/simplejson#347

* Documented RawJSON usage with examples and caveats
  simplejson/simplejson#346

* Added pyproject.toml for PEP 517 build support. setup.py is retained
  for Python 2.7 wheel builds and backwards compatibility.

* Migrated build_ext import from distutils to setuptools in setup.py.
  The distutils.errors imports are kept since setuptools vendors
  distutils on Python 3.12+ where stdlib distutils was removed.

* CI now tests PEP 517 builds (pyproject.toml) alongside the existing
  setup.py-based builds.

* Added Pyodide (wasm32) wheel builds with C speedups via cibuildwheel.
  Previously Pyodide users fell back to the pure-Python wheel; now they
  get the compiled C extension cross-compiled to WebAssembly. Thread
  and subprocess tests are skipped on Emscripten where those APIs are
  unavailable.

* Test suite now fails (instead of skipping) when C speedups are missing
  during cibuildwheel runs, catching broken extension builds early.

* New ``array_hook`` parameter for ``loads()``, ``load()``, and
  ``JSONDecoder``. Called with each decoded JSON array (as a list),
  its return value replaces the list. Analogous to ``object_hook``
  for dicts. Works with both the Python decoder and C scanner.
  (Matches CPython 3.15 json module.)

* Trailing comma detection: the decoder now raises ``JSONDecodeError``
  with "Illegal trailing comma before end of object/array" for inputs
  like ``[1,]`` and ``{"a": 1,}`` instead of generic error messages.
  Both the Python decoder and C scanner are updated.
  (Matches CPython 3.13+ json module.)

* ``frozendict`` encoding support: when ``frozendict`` is available
  (CPython 3.15+ PEP 814), it is encoded as a JSON object just like
  ``dict``. No effect on older Python versions.

* Serialization errors now include ``add_note()`` context on Python
  3.11+ (PEP 678), annotating exceptions with the path to the error,
  e.g. "when serializing list item 1" / "when serializing dict item
  'key'". Only applies to the Python encoder.

* New C fast path for ``encode_basestring`` (``ensure_ascii=False``).
  Previously the non-ASCII string encoder fell back to pure Python;
  now it has a C implementation matching the existing
  ``encode_basestring_ascii`` fast path.
  simplejson/simplejson#207

* The Python decoder now rejects non-ASCII digits (e.g. fullwidth
  ``\uff10``) in JSON numbers, matching the C scanner behavior.
  The ``NUMBER_RE`` regex was changed from ``\d`` to ``[0-9]``.

* Removed dead single-phase init code for Python 3.3/3.4 from the
  C extension (these versions are no longer supported).
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request May 21, 2026
Version 4.0.1 released 2026-04-18

* Skip uploading Pyodide/wasm wheels to PyPI, which rejects them with
  "unsupported platform tag 'pyodide_2024_0_wasm32'". The wheels are
  still built in CI and preserved as workflow artifacts.
  simplejson/simplejson#375

Version 4.0.0 released 2026-04-18

* simplejson 4 requires Python 2.7 or Python 3.8+. Older Python
  versions (2.5, 2.6, 3.0-3.7) are no longer supported. pip will
  not install simplejson 4 on unsupported versions.

* The C extension now uses heap types and per-module state instead of
  static types and global state. This is required for free-threading
  support and sub-interpreter isolation. The Python-level API is
  unchanged.

* Full support for Python 3.13+ free-threading (PEP 703). The C
  extension is now safe to use with the GIL disabled (python3.14t):
  - Converted all static types to heap types with per-module state
  - Added per-object critical sections to scanner and encoder
  - Added free-threading-safe dict operations for Python 3.13+
  - Unified per-module state management and templated parser
  simplejson/simplejson#363
  simplejson/simplejson#364
  simplejson/simplejson#365
  simplejson/simplejson#367
  simplejson/simplejson#369

* Numerous C extension memory safety fixes:
  - Fix use-after-free and leak in encoder ident handling
  - Fix NULL dereferences on OOM in module init and static string init
  - Fix reference leaks in dict encoder (skipkeys item, variable shadowing)
  - Fix member table copy-paste, exception clobbering, missing Py_VISIT
  - Fix error-as-truthy bugs in maybe_quote_bigint and is_raw_json
  - Fix iterable_as_array swallowing MemoryError and KeyboardInterrupt
  - Fix for_json and _asdict swallowing MemoryError, KeyboardInterrupt,
    and other non-AttributeError exceptions raised by user __getattr__
  simplejson/simplejson#355
  simplejson/simplejson#356
  simplejson/simplejson#357
  simplejson/simplejson#358
  simplejson/simplejson#359
  simplejson/simplejson#360
  simplejson/simplejson#373

* C/Python parity fixes:
  - Fix C scanstring off-by-one bounds checks that caused truncated
    or boundary \uXXXX escapes to raise "Invalid \\uXXXX escape
    sequence" instead of "Unterminated string", and report error
    position at the 'u' instead of the leading backslash. The C and
    Python decoders now agree on exception class, message, and
    position across all tested edge cases.
  - Align the Python encoder's dispatch order with the C encoder for
    objects that define _asdict(). Previously a list/tuple/dict
    subclass with an _asdict() method encoded as its container type
    under the Python encoder and as the _asdict() return value under
    the C encoder; both now check _asdict() before list/tuple/dict.
    for_json() continues to outrank _asdict() in both.
  - Fix C scanstring raising a plain ValueError ("end is out of
    bounds") instead of JSONDecodeError for out-of-range end indices.
    User code with `except JSONDecodeError:` now catches both the
    C and pure-Python paths consistently.
  simplejson/simplejson#372

* C extension performance and correctness improvements:
  - Add PyDict_Next fast path for unsorted exact-dict encoding,
    avoiding intermediate items list and N tuple allocations
  - Add indexed fast path for exact list/tuple encoding, avoiding
    iterator allocation and per-item PyIter_Next overhead
  - Use PyUnicodeWriter as JSON_Accu backend on Python 3.14+,
    eliminating intermediate string objects and ''.join calls
  - Fix integer overflow in ascii_escape output_size calculation
    that could cause buffer overwrite on pathologically large strings
  - Fix list encoder separator counter overflow (int to Py_ssize_t)
  - Dead code cleanup (unreachable NULL checks, do-while wrappers)
  simplejson/simplejson#370

* Added Python 3.14 support and updated to cibuildwheel 3.2.1. CI now
  tests free-threaded (3.14t) and debug builds with -Werror, refcount
  leak detection, and GIL-disabled mode.
  simplejson/simplejson#343

* Added a ThreadSanitizer (TSan) stress test CI job. Builds a
  TSan-instrumented free-threaded CPython (cached between runs) and
  runs a concurrent stress test script against the C extension to
  catch data races under free-threading.
  simplejson/simplejson#373

* Replace deprecated license classifiers with SPDX license expression
  simplejson/simplejson#347

* Documented RawJSON usage with examples and caveats
  simplejson/simplejson#346

* Added pyproject.toml for PEP 517 build support. setup.py is retained
  for Python 2.7 wheel builds and backwards compatibility.

* Migrated build_ext import from distutils to setuptools in setup.py.
  The distutils.errors imports are kept since setuptools vendors
  distutils on Python 3.12+ where stdlib distutils was removed.

* CI now tests PEP 517 builds (pyproject.toml) alongside the existing
  setup.py-based builds.

* Added Pyodide (wasm32) wheel builds with C speedups via cibuildwheel.
  Previously Pyodide users fell back to the pure-Python wheel; now they
  get the compiled C extension cross-compiled to WebAssembly. Thread
  and subprocess tests are skipped on Emscripten where those APIs are
  unavailable.

* Test suite now fails (instead of skipping) when C speedups are missing
  during cibuildwheel runs, catching broken extension builds early.

* New ``array_hook`` parameter for ``loads()``, ``load()``, and
  ``JSONDecoder``. Called with each decoded JSON array (as a list),
  its return value replaces the list. Analogous to ``object_hook``
  for dicts. Works with both the Python decoder and C scanner.
  (Matches CPython 3.15 json module.)

* Trailing comma detection: the decoder now raises ``JSONDecodeError``
  with "Illegal trailing comma before end of object/array" for inputs
  like ``[1,]`` and ``{"a": 1,}`` instead of generic error messages.
  Both the Python decoder and C scanner are updated.
  (Matches CPython 3.13+ json module.)

* ``frozendict`` encoding support: when ``frozendict`` is available
  (CPython 3.15+ PEP 814), it is encoded as a JSON object just like
  ``dict``. No effect on older Python versions.

* Serialization errors now include ``add_note()`` context on Python
  3.11+ (PEP 678), annotating exceptions with the path to the error,
  e.g. "when serializing list item 1" / "when serializing dict item
  'key'". Only applies to the Python encoder.

* New C fast path for ``encode_basestring`` (``ensure_ascii=False``).
  Previously the non-ASCII string encoder fell back to pure Python;
  now it has a C implementation matching the existing
  ``encode_basestring_ascii`` fast path.
  simplejson/simplejson#207

* The Python decoder now rejects non-ASCII digits (e.g. fullwidth
  ``\uff10``) in JSON numbers, matching the C scanner behavior.
  The ``NUMBER_RE`` regex was changed from ``\d`` to ``[0-9]``.

* Removed dead single-phase init code for Python 3.3/3.4 from the
  C extension (these versions are no longer supported).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Minor: member table copy-paste bug, exception clobbering, missing Py_VISIT

2 participants