Handle non-finite Decimals like floats in the encoder (fixes #149) by apoorvdarshan · Pull Request #381 · simplejson/simplejson

apoorvdarshan · 2026-07-09T15:45:15Z

Summary

Fixes #149.

Non-finite Decimal values were not handled consistently with the
matching float. Both the pure-Python encoder and the C extension
serialized a Decimal by calling str() on it directly, bypassing the
allow_nan / ignore_nan handling that floats go through. The result
was invalid JSON:

>>> import simplejson, math
>>> from decimal import Decimal
>>> simplejson.dumps([Decimal('0.33'), Decimal('NaN'), Decimal('0.20')], ignore_nan=True)
'[0.33, NaN, 0.20]'                 # before: invalid; float gives [..., null, ...]
>>> simplejson.dumps(Decimal('NaN'), allow_nan=False)
'NaN'                               # before: invalid; float raises ValueError

Fix

Non-finite Decimal values (NaN, sNaN, Infinity, -Infinity) are
now routed through the same float NaN/Inf handling (floatstr /
encoder_encode_float) as floats, so they follow allow_nan and
ignore_nan identically:

mode	non-finite Decimal	matches float
`allow_nan=True`	`NaN` / `Infinity` / `-Infinity`	yes
`ignore_nan=True`	`null`	yes
`allow_nan=False` (default)	raises `ValueError`	yes

Finite Decimal values are left completely untouched — they are still
stringified as-is, so precision and trailing zeros (0.00, 1.50,
-0) are preserved. This is why the fix keys on is_finite() rather
than the is_normal() suggestion from the issue thread: is_normal()
is False for 0, 0.00 and subnormals, which would have routed them
through float() and dropped their trailing zeros.

float(Decimal('sNaN')) raises, so signaling NaNs are mapped to a plain
NaN before going through the float path.

C path

The bug reproduces on both the C extension and the pure-Python
encoder (Decimal encoding goes through whichever encoder is active), so
the fix is applied to both. The C side adds a small
encoder_encode_decimal helper that mirrors the Python logic, using
cached interned is_finite / is_nan attribute names in the module
state (matching the existing JSON_attr_* pattern) with proper
GC traverse/clear wiring.

Tests

Added regression tests to simplejson/tests/test_decimal.py covering
allow_nan=True / ignore_nan=True / allow_nan=False for non-finite
Decimals as top-level values, container values and dict keys, plus a
test_decimal_finite_unaffected guard confirming finite Decimals are
unchanged. The new tests fail on main and pass with the fix.

Verified locally with the _cibw_runner harness, which runs the full
suite twice — once with the C speedups loaded and once via
NoExtensionTestSuite (pure-Python): 458 tests pass on both paths, and
the extension wiring check (c_make_encoder is make_encoder) holds.
The C extension builds clean under the project's
-Werror -Wdeclaration-after-statement CFLAGS.

Also added a CHANGES.txt entry (under a provisional 4.1.2 (unreleased) heading — happy to adjust the version/date to whatever
you prefer).

Disclosure: prepared with AI assistance; reviewed and verified locally.

Non-finite Decimal values (NaN, sNaN, Infinity) bypassed the allow_nan / ignore_nan handling that floats go through, so ignore_nan=True emitted a literal NaN instead of null and allow_nan=False silently produced invalid JSON instead of raising ValueError. Route non-finite Decimals through the same float NaN/Inf handling in both the pure-Python encoder and the C extension. Finite Decimals (including trailing-zero forms such as 0.00) are left untouched, so precision is preserved. Signaling NaN, which cannot be converted with float(), is mapped to a plain NaN. Fixes simplejson#149.

etrepum

It would probably be better to unconditionally convert to string and then check the first one or two characters to see if it’s a special case

Adopt the maintainer's suggested approach: unconditionally stringify the Decimal once and inspect the first (or first-two) characters to detect the non-finite case, rather than calling is_finite()/is_nan(). str(Decimal) carries a leading '-' at most, so a finite value's first significant char is always a digit while a non-finite value's is a letter (N/s for [s]NaN, I for Infinity). This drops the two interned attribute names (is_finite/is_nan), their GC traverse/clear wiring and the method-call helper from the C extension, leaving encoder_encode_decimal a self-contained str()+char-check. Behaviour is unchanged and identical across the C and pure-Python paths.

apoorvdarshan · 2026-07-09T16:59:27Z

Good call — switched to that approach in 0dcb58c. Both the pure-Python encoder and the C extension now stringify the Decimal once and branch on the first (or first-two, after a leading -) characters: a finite value's first significant char is always a digit, while a non-finite one is a letter (N/s for [s]NaN, I for Infinity). Non-finite values are mapped to the matching float and routed through the existing NaN/Inf handling; finite values are emitted verbatim.

This also let me drop the two interned attr names (is_finite/is_nan), their GC traverse/clear wiring and the method-call helper from _speedups.c, so encoder_encode_decimal is now a self-contained str() + char check. Behavior is unchanged and byte-identical across the C and pure-Python paths (verified via _cibw_runner, 458 tests on both paths; the extension builds clean under -Werror -Wdeclaration-after-statement).

etrepum · 2026-07-09T17:28:18Z

+    c = s[1] if s[0] == '-' else s[0]
+    if c <= '9':


Suggested change

c = s[1] if s[0] == '-' else s[0]

if c <= '9':

c0 = s[:1]

c = s[1:2] if c0 == '-' else c0

if '0' <= c <= '9':

Might be worth being a little defensive here in case Decimal is subclassed in a strange way

etrepum · 2026-07-09T17:29:11Z

+    c = (Py_UCS4)0;
+    if (len > (negative ? 1 : 0))
+        c = PyUnicode_READ_CHAR(str, negative ? 1 : 0);
+    if (c <= '9') {


Suggested change

if (c <= '9') {

if (IS_DIGIT(c)) {

etrepum · 2026-07-09T17:32:28Z

@@ -1,3 +1,15 @@
+Version 4.1.2 (unreleased)


I update the changelog, you can remove this

etrepum · 2026-07-09T17:37:02Z

 }

+static PyObject *
+encoder_encode_decimal(PyEncoderObject *s, PyObject *obj)


This breaks Python 2.7 compatibility, there should be a separate branch for that in here since PyObject_Str isn't going to return a unicode object.

_speedups.obj : error LNK2019: unresolved external symbol _PyUnicode_READ_CHAR referenced in function _encoder_encode_decimal build\lib.win32-2.7\simplejson\_speedups.pyd : fatal error LNK1120: 1 unresolved externals

etrepum · 2026-07-09T17:42:50Z

+    if (c == 'I') {
+        floatobj = PyFloat_FromDouble(negative ? -Py_HUGE_VAL : Py_HUGE_VAL);
+    }
+    else {
+        floatobj = PyFloat_FromDouble(Py_NAN);
+    }
+    if (floatobj == NULL)
+        return NULL;
+    encoded = encoder_encode_float(s, floatobj);


these should just go through the same path that encoder_encode_float does without using a float, e.g. incref and return the correct state->JSON_* value

etrepum reviewed Jul 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Handle non-finite Decimals like floats in the encoder (fixes #149)#381

Handle non-finite Decimals like floats in the encoder (fixes #149)#381
apoorvdarshan wants to merge 2 commits into
simplejson:mainfrom
apoorvdarshan:fix-decimal-nan-149

apoorvdarshan commented Jul 9, 2026

Uh oh!

etrepum left a comment

Uh oh!

apoorvdarshan commented Jul 9, 2026

Uh oh!

etrepum Jul 9, 2026

Uh oh!

etrepum Jul 9, 2026

Uh oh!

etrepum Jul 9, 2026

Uh oh!

etrepum Jul 9, 2026

Uh oh!

etrepum Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-    c = s[1] if s[0] == '-' else s[0]
-    if c <= '9':
+    c0 = s[:1]
+    c = s[1:2] if c0 == '-' else c0
+    if '0' <= c <= '9':

Uh oh!

Conversation

apoorvdarshan commented Jul 9, 2026

Summary

Fix

C path

Tests

Uh oh!

etrepum left a comment

Choose a reason for hiding this comment

Uh oh!

apoorvdarshan commented Jul 9, 2026

Uh oh!

etrepum Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

etrepum Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

etrepum Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

etrepum Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

etrepum Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants