support msgpack encoding with integer keys by jensbjorgensen · Pull Request #1119 · msgspec/msgspec

jensbjorgensen · 2026-07-10T16:21:41Z

Add int_keys: integer MessagePack map keys for Structs

Motivation

There's an inherent tension between having descriptive field names on a Struct and
the size of its encoded representation — every field name is repeated, in full, in
every message. Since one of msgspec's biggest draws is its encoding efficiency,
being able to encode a Struct's fields with integer keys instead of their names is
an attractive way to shrink MessagePack messages further.

rename to single-character names helps, and is the right tool when you need a
readable/valid JSON representation. But MessagePack can do better: it supports
integer map keys directly, and even in JavaScript the common MessagePack decoders
handle integer keys gracefully. This PR makes that available.

What this adds

A new Struct configuration option, int_keys, mapping field names to integers:

class Point(msgspec.Struct, int_keys={"x": 1, "y": 2}):
    x: int
    y: int

>>> msgspec.msgpack.decode(msgspec.msgpack.encode(Point(1, 2)))
{1: 1, 2: 2}
>>> msgspec.msgpack.decode(msgspec.msgpack.encode(Point(1, 2)), type=Point)
Point(x=1, y=2)

Design

MessagePack-only. JSON object keys must be strings, so the JSON encoder
ignores int_keys and keeps using the (possibly renamed) field names — no error,
and the existing fast JSON path is untouched.
A dedicated option, not an extension of rename. rename is string-valued
and applies to both codecs; keeping int_keys separate avoids overloading its
meaning, and the two compose cleanly (a field can be renamed for JSON and
int-keyed for MessagePack).
Reuses existing machinery. Encoding integer keys uses mpack_encode_long;
decoding uses the same IntLookup reverse-lookup that already backs integer
union tags — so this leans on well-trodden code paths rather than introducing new
ones.
Minimal overhead when unused. The two new StructMetaObject members are NULL
for Structs without int_keys, and the encode/decode fast paths are unchanged for
them. On decode, the key type is only inspected for Structs that opt in.
Baked onto the type. Nested Structs are handled automatically — each encodes
with its own keys, with nothing extra required at encode/decode time. Partial maps
are allowed (unlisted fields keep their string keys); keys must be unique within a
Struct and fit in a signed 64-bit integer.
Introspectable. The assigned key is exposed via
msgspec.structs.fields(...).int_key.

Compatibility, tests, docs

Fully backward compatible — no behavior change for Structs that don't use
int_keys. Includes tests (tests/unit/test_msgpack.py, tests/unit/test_struct.py)
and documentation (docs/structs.rst).

Please consider this addition to msgspec — I'm eager to discuss any amendments or
adjustments that would help it fit the project.

Siyet

Thanks for the thorough PR, and for building the decode path on the existing IntLookup machinery from #135, that's the right foundation.

One correctness blocker

An int-keyed struct that is also tagged decodes fine as a concrete type, but breaks as a union member whenever the string tag is not the first key on the wire. This reproduces entirely in-library through the sorted encoder, which orders int keys ahead of the string tag:

class A(msgspec.Struct, tag=True, int_keys={"a": 1, "b": 2}):
    a: int
    b: int

class B(msgspec.Struct, tag=True, int_keys={"a": 1}):
    a: int

buf = msgspec.msgpack.Encoder(order="sorted").encode(A(1, 2))  # {1:1, 2:2, "type":"A"}
msgspec.msgpack.decode(buf, type=A | B)
# ValidationError: Expected `str`, got `int` - at `key` in `$`

So msgspec's own sorted encoder emits bytes its own union decoder then rejects. mpack_decode_struct_union scans map keys with mpack_decode_cstr to find the tag and can't skip past an int key to reach it; string-keyed unions are order-independent precisely because it can skip string keys, so this is a regression relative to them. A foreign producer that doesn't put the tag first hits the same wall. The test_sorted_order_tagged test only decodes as a concrete type, so CI stays green over the hole.

Fix needs one of: int-key handling in the union tag-scan, a guarantee the tag encodes first even under order="sorted", or an explicit tag + int_keys restriction. Whichever way, this path wants a union test.

(No memory-safety issues found in the C changes on a careful read: refcounting, GC traversal/clear, NULL-checks, the class-creation validation, and the sorted-path recursion/alloc handling are all balanced. The blocker above is a logic/interaction bug, not a leak.)

Two silent no-op edges worth a decision

int_keys together with array_like=True silently drops int_keys (array_like wins), with no error at class creation and no test covering the combo.
int_keys is silently ignored on JSON.

Both are documented and consistent (no crash, no round-trip mismatch), but neither gives the user any signal.

Scope / design, for a maintainer call

The motivation is smaller MessagePack, but array_like=True is already strictly smaller (no keys at all) and applies to both codecs, and rename to short names covers the readable-and-evolvable case on both codecs. The genuine niche int_keys fills is compact and schema-evolvable (protobuf-style stable field numbers: order-independent, tolerant of gaps, sparse with omit_defaults) which is real but narrow, and the PR doesn't spell it out. Separately, the MessagePack-only-with-silent-JSON-fallback framing cuts against how integer dict keys were handled in #243, where they were made to work in JSON via string coercion rather than being codec-specific.

This is new public API surface (a class kwarg + FieldInfo.int_key + a new __struct_encode_int_keys__ dunder) in the deliberately-small Struct-config area, so it deserves a design sign-off before it moves forward. @jcrist @provinzkraut could one of you weigh in on whether the niche justifies a standalone option, and on the JSON asymmetry? Happy to do a full correctness/test pass once the shape is settled.

…ion-tagging correctness

jensbjorgensen · 2026-07-17T16:53:41Z

Thanks for the careful review — especially confirming the C changes are memory-clean.

Correctness blocker (fixed)

Fixed by teaching the union tag-scan to skip non-string keys. mpack_decode_struct_union
now peeks each key's type and skips any non-string key (e.g. an integer key from an
int_keys struct) while looking for the string tag field, so the tag is found regardless
of its position — the same order-independence string-keyed unions already have. The
concrete-type decode path already handled int keys via IntLookup; only the union scan
needed to become key-type-aware. Nothing changes for string-keyed unions.

Added union tests:

test_tagged_union_decode_order_independent — decodes A | B for both the default and
order="sorted" encoders (the sorted case is exactly your repro: {1:1, 2:2, "type":"A"}).
test_tagged_union_decode_tag_last — a foreign-producer layout with the tag as the last
key.

array_like + int_keys (now errors)

int_keys together with array_like=True now raises ValueError at class creation
("array-encoded structs have no field keys") rather than silently dropping int_keys.
Test added (test_int_keys_array_like_rejected, class + defstruct forms).

JSON asymmetry (now implemented via string coercion)

Rather than leave the asymmetry, we implemented the #243-style approach, so int_keys is
now codec-agnostic:

Encode: JSON object keys are the integer ids coerced to decimal strings, e.g.
{"1": 3, "2": 4} (msgpack uses native integer keys, {1: 3, 2: 4}). This holds under
order='sorted' too — the sorted JSON output emits the int-string keys in the same order
as the sorted msgpack output, so the two codecs stay consistent.
Decode: a JSON string key is resolved by parsing it as an integer and looking it up in the
struct's int-key table; if it isn't a mapped integer, we fall back to matching it as a
field name, so a name-keyed (e.g. foreign, or upstream-renamed) producer still round-trips.

No overhead for structs without int_keys (the JSON encode/decode paths only branch when
the type has an int-key table). Tests added for JSON encode/decode, partial maps, tagged
unions in JSON, and the sorted path (asserting it emits int-string keys, tag included).

Happy to change the shape (back to msgpack-only, or an explicit error on JSON) if you'd
rather — but this removes the asymmetry and keeps int_keys consistent with how #243 handled
integer dict keys.

Scope / the niche it fills

Fair point that the motivation undersold this. The niche isn't just "smaller" — it's
compact and schema-evolvable, i.e. protobuf-style stable field numbers:

array_like=True is smaller still, but positional — you can't reorder or remove a field,
there are no gaps, and it's not tolerant of producer/consumer schema skew.
rename to short names stays evolvable but keeps string keys (larger, and single-char
names collide fast).
int_keys is the intersection: order-independent, gap-tolerant (retired ids are never
reused), and sparse under omit_defaults — a field can be added/removed/omitted without
breaking existing readers, while staying as compact as short integer keys allow. That's
the case neither existing option covers, and it's what makes it worth a stable per-field id
map rather than positional or string encoding.

Happy to fold this framing into the PR description, and to do the full correctness/test pass
once the shape (standalone option vs. not, and the JSON question) is settled.

jensbjorgensen · 2026-07-28T23:32:05Z

Hello, just checking in, wondering if anyone maybe had a look at the updates?

add int_keys constructor option

a981a42

jensbjorgensen had a problem deploying to docs-preview July 10, 2026 16:21 — with GitHub Actions Failure

Siyet reviewed Jul 13, 2026

View reviewed changes

fix array-like conflict (exception), align json handling with 243, un…

0985b52

…ion-tagging correctness

jensbjorgensen had a problem deploying to docs-preview July 17, 2026 16:47 — with GitHub Actions Failure

update documentation to reflect changes up to the present

9a72361

jensbjorgensen had a problem deploying to docs-preview July 17, 2026 17:27 — with GitHub Actions Failure

Navigation Menu

Uh oh!

support msgpack encoding with integer keys - #1119

support msgpack encoding with integer keys#1119
jensbjorgensen wants to merge 3 commits into
msgspec:mainfrom
jensbjorgensen:msgpack_intkeys

jensbjorgensen commented Jul 10, 2026

Uh oh!

Siyet left a comment

Uh oh!

jensbjorgensen commented Jul 17, 2026

Uh oh!

jensbjorgensen commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jensbjorgensen commented Jul 10, 2026

Motivation

What this adds

Design

Compatibility, tests, docs

Uh oh!

Siyet left a comment

Choose a reason for hiding this comment

One correctness blocker

Two silent no-op edges worth a decision

Scope / design, for a maintainer call

Uh oh!

jensbjorgensen commented Jul 17, 2026

Correctness blocker (fixed)

array_like + int_keys (now errors)

JSON asymmetry (now implemented via string coercion)

Scope / the niche it fills

Uh oh!

jensbjorgensen commented Jul 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants