Support numeric constraints for decimal.Decimal by Siyet · Pull Request #1006 · msgspec/msgspec

Siyet · 2026-04-09T19:16:26Z

Reopening #998 (auto-closed when fork was deleted).

Fixes #683.

Previously, the numeric constraints gt, ge, lt, le, and multiple_of in msgspec.Meta were only valid for int and float types. This PR extends support to decimal.Decimal.

PR #692 by @cocorigon has been open since May 2024. The author has since explicitly declined to continue it and invited others to take it over. This is a fresh implementation with documentation and full test coverage.

Changes

_core.c: Added five new TypeNode constraint flags backed by PyObject* slots storing Decimal instances. A new ms_check_decimal_constraints() function is called after every Decimal decode path.
__init__.pyi: Updated Meta.__init__ parameter types to include Decimal.
docs/constraints.rst: Updated Numeric Constraints section to mention decimal.Decimal.
tests/unit/test_constraints.py: Added TestDecimalConstraints covering all constraint types.

Notable design decisions

multiple_of for Decimal uses Python's % operator, which gives exact arithmetic.
Constraint bounds passed as float are converted to Decimal via string representation to preserve precision.

ofek · 2026-04-09T21:32:40Z

It looks like the Git history needs to be fixed on this one.

Siyet · 2026-04-10T12:08:25Z

The failing build job here is unrelated to this PR: it's the link checker tripping on the Pydantic docs redirect (docs.pydantic.dev/latest/ → pydantic.dev/docs/validation/...), which is fixed in #1008. Once #1008 lands and this branch is rebased, CI should go green.

Siyet · 2026-04-10T12:28:35Z

@ofek done — force-pushed a clean rewrite of the branch:

Squashed into a single commit with a proper message and Closes #683 reference.
Rebased onto current main, dropping all the leftover commits from the deleted fork (CODEOWNERS, package rename, doc-link rewrites, plus changes that belong to other PRs).
Scope is now strictly the four Decimal-relevant files: _core.c, __init__.pyi, docs/constraints.rst, tests/unit/test_constraints.py. Diff stat is +353/-52.
MS_CONSTR_DECIMAL_* flags claim bits 36-40, the lowest unused slots in TypeNode->types. If Support Literal[True] and Literal[False] types #1004 lands first, I'll rebase to whatever bits are next free.

Build job is still red on the link checker but that's the unrelated #1008 fix; nothing in this branch touches it.

Extends gt, ge, lt, le, and multiple_of Meta constraints to work with decimal.Decimal types, enabling exact arithmetic comparisons and sub-integer precision constraints without floating point issues. Constraint bounds passed as int or float are coerced to Decimal via their string representation to preserve precision. multiple_of uses Python's % operator for exact arithmetic. Closes #683

Siyet · 2026-04-20T19:22:46Z

Rebased onto main now that #1004 has landed. #1004 took bits 36-37 for MS_TYPE_BOOLLITERAL_TRUE/FALSE, so I shifted the Decimal constraint flags to the next free slots:

MS_CONSTR_DECIMAL_GT/GE/LT/LE -> bits 38-41 (the remaining gap between types and existing constraints)
MS_CONSTR_DECIMAL_MULTIPLE_OF -> bit 61 (the only free slot that's left below the reserved MS_EXTRA_FLAG at 63)

No other constraint bits were moved. SLOT numbering and the TypeNode->details[] ordering are unchanged (SLOTs use the named constants, not raw bit positions).

provinzkraut

I think this should have a check that disallows Decimal for gt, ge, etc. constraints where the type is not also a Decimal

Coercing a Decimal bound to int or float silently drops precision, which defeats the point of specifying the bound as a Decimal in the first place. Only allow Decimal bounds when the annotated type is also Decimal, and raise a TypeError otherwise at TypeNode construction time.

Siyet · 2026-04-21T07:25:47Z

@provinzkraut added the check: a Decimal in gt/ge/lt/le/multiple_of now raises TypeError at encoder construction time when the annotated type isn't Decimal. Coercing to int/float loses precision, so a Decimal bound there is pointless.

Covered with a parametrized test, docs updated.

sobolevn · 2026-06-15T10:05:13Z

-        lt: Union[int, float, None] = None,
-        le: Union[int, float, None] = None,
-        multiple_of: Union[int, float, None] = None,
+        gt: Union[int, float, Decimal, None] = None,


This can now be a type alias, since it is repeated quite a lot and can be changed.

Done, pulled it out into _NumericBound = int | float | Decimal | None.

sobolevn · 2026-06-15T10:07:44Z

            return false;
        }
    }
+    else if (PyObject_IsInstance(val, mod->DecimalType)) {


This allows Decimal subclasses, but you never test them. Since int and float subclasses are not allowed above - what is the correct way to handle this?

int/float use *_CheckExact because bool is an int subclass (we don't want to silently accept True as a numeric bound). Decimal has no such footgun subclass, and _constr_as_decimal normalizes the bound through Decimal(obj) at TypeNode-build time anyway, so a subclass's overridden dunders never reach the checks (verified with a hostile subclass overriding __gt__/__mod__ to raise: they're never called). Accepting subclasses is intentional; added test_decimal_subclass_bound.

sobolevn · 2026-06-15T10:40:13Z

        if ((out = TypeNode_traverse(node, visit, arg)) != 0) return out;
    }
+    /* Traverse Decimal constraint PyObject* */
+    if (self->types & (MS_CONSTR_DECIMAL_GT | MS_CONSTR_DECIMAL_GE)) {


This should be done before return out. Otherwise it can be left in a broken state.

Agreed, moved the Decimal-constraint Py_VISITs before the loops that can return out early. They're direct references on this node, so visiting them before recursing into children is the right place. (The Decimal pointers aren't counted in n_obj, so there's no double-visit.)

sobolevn · 2026-06-15T10:45:28Z

+        PyObject *c = TypeNode_get_constr_decimal_min(type);
+        int ok = PyObject_RichCompareBool(obj, c, Py_GT);
+        if (ok == -1) return false;
+        if (MS_UNLIKELY(!ok)) {


I don't think that it is unlikely. I would say that ok == -1 is unlikely.

The existing ms_decode_constr_int (and the float/str checks) mark the validation-failure path !ok as MS_UNLIKELY since valid data is the hot path, so I kept that for consistency. You're right about the error path though: marked ok < 0 as MS_UNLIKELY (and switched == -1 to < 0) in all four comparison blocks.

sobolevn · 2026-06-15T10:46:05Z

+        PyObject *modulo = PyNumber_Remainder(obj, c);
+        if (modulo == NULL) return false;
+        PyObject *zero = PyLong_FromLong(0);
+        if (zero == NULL) { Py_DECREF(modulo); return false; }


Suggested change

if (zero == NULL) { Py_DECREF(modulo); return false; }

if (zero == NULL) {

Py_DECREF(modulo);

return false;

}

style nit.

Dropped this block entirely, there's no zero to allocate anymore (see the multiple_of comment below).

sobolevn · 2026-06-15T10:48:13Z

+        PyObject *c = TypeNode_get_constr_decimal_multiple_of(type);
+        PyObject *modulo = PyNumber_Remainder(obj, c);
+        if (modulo == NULL) return false;
+        PyObject *zero = PyLong_FromLong(0);


You don't need to do the second compare in Python, you can do this in C.

Convert modulo to PyNumber_AsSsize_t (or whatever) and compare it with == 0 :)

PyNumber_AsSsize_t won't work here: Decimal has no __index__, so it'd raise TypeError on any remainder (including integral ones). Switched to PyObject_IsTrue(modulo) instead, a Decimal is falsy iff it's zero (including -0, 0E+10). That drops the zero allocation and stays correct for fractional remainders like Decimal("0.5").

If we allow Decimal subclasses, we can't use PyObject_IsTrue, because __bool__ can be changed in subclasses :(

Good point, switched to an explicit PyObject_RichCompareBool(modulo, zero, Py_EQ) against a cached small int - it expresses the intent directly and doesn't depend on __bool__ at all.

For the record, I checked reachability while fixing this: the scenario can't actually trigger today. Subclass values never reach the check (convert rejects them with Expected `decimal`, got `WeirdDecimal` , and decode paths construct plain Decimal), subclass bounds are normalized via Decimal(bound) at collection, and decimal arithmetic returns base-class instances anyway, so the remainder was always a plain Decimal. Still, the explicit comparison is clearly better here, and there's now a test pinning the behavior with a __bool__-overriding subclass bound (test_multiple_of_decimal_subclass_bound_bool_override).

sobolevn · 2026-06-15T10:50:50Z

+
+    `decimal.Decimal` bounds (``gt``, ``ge``, ``lt``, ``le``, ``multiple_of``)
+    are only valid on `decimal.Decimal` types. Applying a `decimal.Decimal`
+    bound to an ``int`` or ``float`` type raises a `TypeError`, since coercing


Sorry, I have a question about int here. How does it lose precision?

Good catch, "lose precision" was imprecise for int. It's not truncation: the bound would be converted via PyLong_AsLongLongAndOverflow, which needs __index__, and Decimal has none, so any Decimal (Decimal("2") as well as Decimal("1.5")) is rejected by the int conversion, and a fractional one has no integer equivalent at all. For float the bound would be rounded to the nearest binary float. Reworded the note to drop the incorrect "truncate".

# Conflicts: # src/msgspec/__init__.pyi # tests/unit/test_constraints.py

Siyet · 2026-06-15T23:31:26Z

@provinzkraut this is already handled in this PR: a Decimal bound on an int/float type is rejected with a TypeError (typenode_collect_constraints, via PyObject_IsInstance(..., DecimalType)), covered by test_invalid_decimal_bound_on_non_decimal for all five of gt/ge/lt/le/multiple_of. The check runs at TypeNode construction, so it's protocol-agnostic (msgpack behaves identically).

Frees up 4 bits in the types bitmask (main now uses bits 38-39 for frozendict, leaving too few free bits for per-constraint decimal bits). Also compare the multiple_of remainder against zero explicitly instead of relying on __bool__.

MS_TYPE_FROZENDICT took bits 38-39, MS_CONSTR_DECIMAL moved to bit 40.

Siyet · 2026-07-02T13:07:27Z

Rebased on main and reworked the constraint storage, two commits:

Storage redesign (38f7e7c): decimal constraints now live in a single details slot holding a 5-tuple (gt, ge, lt, le, multiple_of) (None for unset), gated by one MS_CONSTR_DECIMAL bit. This was forced by bit-space math: Add frozendict support for 3.15+ #1052 took bits 38-39 for frozendict, leaving only 4 free bits (40, 41, 61, 62) in the types mask while the old per-constraint layout needed 5. Sharing bits with the float constraints isn't an option since constrained members of different numeric types can coexist in one union (test_mix_float_and_int). The single-slot shape follows the literal-lookup precedent (one bit, one object holding the details). Validation is unchanged behavior-wise; the check is already a cold path gated by one bit test.
Merge main (def7a7b): trivial after the redesign, MS_CONSTR_DECIMAL landed on bit 40.

Also addressed the PyObject_IsTrue review comment: the remainder is now compared against zero explicitly (see inline reply), with a regression test for a __bool__-overriding subclass.

Verified locally on 3.12 and 3.15.0a7 (WSL): full unit suite green on both (6351 / 6439 passed), decimal constraints + frozendict coexist correctly in one union on 3.15 (slot ordering with bits 38-39 + 40), and 100k decoder create/destroy cycles show zero RSS growth (constraint tuple lifecycle is leak-free).

codspeed-hq · 2026-07-02T13:15:37Z

Merging this PR will improve performance by 15.89%

⚡ 3 improved benchmarks
✅ 136 untouched benchmarks
⏩ 135 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	Simulation	`test_decode_type[arm-msgpack-None]`	754.5 µs	623.6 µs	+20.99%
⚡	Simulation	`test_decode_type[arm-json-True]`	944.7 µs	819.2 µs	+15.31%
⚡	Simulation	`test_pickle_load[arm]`	1.8 ms	1.6 ms	+11.55%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing decimal-constraints-683 (def7a7b) with main (4440c96)}

135 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Siyet requested review from jcrist and ofek April 10, 2026 07:32

Siyet mentioned this pull request Apr 10, 2026

Support numeric constraints for Decimal types #692

Closed

Siyet force-pushed the decimal-constraints-683 branch from 8c80626 to f406983 Compare April 10, 2026 12:28

Siyet force-pushed the decimal-constraints-683 branch from d2ce46f to 023b9e1 Compare April 20, 2026 19:22

provinzkraut reviewed Apr 20, 2026

View reviewed changes

sobolevn reviewed Jun 15, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into decimal-constraints-683

b449815

# Conflicts: # src/msgspec/__init__.pyi # tests/unit/test_constraints.py

Siyet temporarily deployed to docs-preview June 15, 2026 22:43 — with GitHub Actions Inactive

Address review feedback on Decimal constraints

72b7932

Siyet temporarily deployed to docs-preview June 15, 2026 23:31 — with GitHub Actions Inactive

Format test_decimal_subclass_bound

f59ed16

Siyet temporarily deployed to docs-preview June 15, 2026 23:39 — with GitHub Actions Inactive

Siyet added 2 commits July 2, 2026 17:00

Merge main (frozendict bit reallocation)

def7a7b

MS_TYPE_FROZENDICT took bits 38-39, MS_CONSTR_DECIMAL moved to bit 40.

Siyet deployed to docs-preview July 2, 2026 13:06 — with GitHub Actions View deployment

Uh oh!

Conversation

Siyet commented Apr 9, 2026

Changes

Notable design decisions

Uh oh!

ofek commented Apr 9, 2026

Uh oh!

Siyet commented Apr 10, 2026

Uh oh!

Siyet commented Apr 10, 2026

Uh oh!

Siyet commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

provinzkraut left a comment

Choose a reason for hiding this comment

Uh oh!

Siyet commented Apr 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Siyet commented Jun 15, 2026

Uh oh!

Siyet commented Jul 2, 2026

Uh oh!

codspeed-hq Bot commented Jul 2, 2026

Merging this PR will improve performance by 15.89%

Performance Changes

Footnotes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Siyet commented Apr 20, 2026 •

edited

Loading