implement multi-phase init by provinzkraut · Pull Request #1024 · msgspec/msgspec

provinzkraut · 2026-04-18T15:12:50Z

Closes #563.

Relatively straightforward, the only slight hiccough is msgspec_get_global_state, which relied on PyState_FindModule / PyState_AddModule, which in turn don't work (well) with multi-phase init. I've replaced them with a global static MsgspecState, set during module init; This is not safe for sub-interpreters, but handling it properly would be a large refactor that I think is best kept for separate PRs.

Road to subinterpreter support

This is the first step for msgspec to support subinterpreters, but it's still a long way to go. I'll open a separate tracking issue for that once this gets merged, but it'll likely involve some larger refactors, especially around moving things to heap types where absolutely necessary, and ensuring this does not impact performance negatively.

provinzkraut · 2026-04-18T15:16:34Z

since we're changing how the subinterpreter support is flagged, i wanted some tests to show it's still working

provinzkraut · 2026-04-18T21:14:43Z

    return 0;
 }

-static struct PyModuleDef msgspecmodule = {


This was simply moved down to make the flow easier, as it references _core_exec

Siyet

Leaving a note for the history / backlink: this obsoletes the PyState_AddModule workaround jcrist added in #561, where the body said "in the long run we should move to using multiphase module init which should avoid this problem entirely." Good to have the thread connected.

goodboy · 2026-04-21T22:50:17Z

I'll open a separate tracking issue for that once this gets merged,

please do ping if/when that happens 🙏🏼

Siyet · 2026-05-26T19:06:54Z

@provinzkraut Happy to push these small fixups to your branch myself if you're short on time, just give me a thumbs up. Keen to get this in so the subinterp work in #1026 can start moving.

provinzkraut · 2026-05-26T20:12:19Z

@Siyet feel free to push. Otherwise I could get it done some time later this week

sobolevn · 2026-06-07T06:45:12Z

+state is populated in _core_exec and cleared and freed by m_clear and m_free
+respectively.
+*/
+static MsgspecState *_core_state = NULL;


I would advokate against this pattern :)
It would be required to refactor it anyway for future FT / SI builds.

Maybe we can instead use something like

static MsgspecState * _get_current_module_state(void) { PyObject *mod = _get_current_module(); if (mod == NULL) { mod = PyImport_ImportModule("msgspec._core"); if (mod == NULL) { return NULL; } } MsgspecState *state = get_module_state(mod); Py_DECREF(mod); return state; }

?

Or PyType_GetModuleByDef / PyType_GetModule :)

I've tried using PyType_GetModule at first, but there were some issues. Can't remember what it was though, it's been a while. Let me re-evaluate that again

PyType_GetModuleByDef is only available on 3.11+. I'll see if PyType_GetModule can be made to work

Best would probably to just pipe the mod through to every caller that needs it. A bit bigger API surface, but less mucking around with global state and such. Currently checking if that's viable

@sobolevn So I played around with it a bit. I tried to reduce the calls to the global state by piping it through wherever possible, which does work. However, it reveals quite a lot of issues, specifically in places where that piping through is currently just not possible where we have static types, which means if we don't want to store process-global state anywhere, things get really complicated (or at least I haven't been able to find a good solution).

What we could do is reverse the implementation order here:

Make the move to heap types

Remove global state

Multi-phase init

The solution I could find to address everything within this PR would be to use change msgspec_get_global_state to use PyInterpreterState_GetDict, but its docs say

This is not a replacement for PyModule_GetState(), which extensions should use to store interpreter-specific state information.

Soo.. I think unless we're fine with having this global state around until we migrate to heap types, I don't see a way how we can get this PR merged. Which is fine, I don't mind doing it the other way around

Oh, we don't have heap types yet 🤦‍♂️
You are right, this is the required first step for new APIs :)

You are right, this is the required first step for new APIs :)

Alright, let's try working towards this then. For that, we really ought to have some continuous benchmarking set up though. I'll take a look at https://codspeed.io

I'm not sure if you can see but I accepted the installation request. It looks like they recommend the OIDC method of authenticating in GitHub Actions and here is the badge:

[![CodSpeed](https://img.shields.io/endpoint?url=https://app.codspeed.io/badge.json)](https://app.codspeed.io/msgspec/msgspec)

sobolevn · 2026-06-07T06:56:38Z

+    .m_methods = msgspec_methods,
+    .m_traverse = msgspec_traverse,
+    .m_clear = msgspec_clear,
+    .m_free =(freefunc)msgspec_free,


Suggested change

.m_free =(freefunc)msgspec_free,

.m_free = (freefunc)msgspec_free,

style nit

sobolevn · 2026-06-07T06:58:42Z

 #ifdef Py_GIL_DISABLED
-    PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED);
+    #if !PY313_PLUS
+        PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED);


This is not needed, we set {Py_mod_gil, Py_MOD_GIL_NOT_USED}, module slot

sobolevn · 2026-06-07T07:01:50Z

+    if (st->astimezone == NULL) return -1;

    /* uuid module imports */
    temp_module = PyImport_ImportModule("uuid");


There are missing Py_DECREF(temp_module) calls in many places.

Yeah, I think this was discussed in a previous PR. I think we should clean those up. Should be a separate PR though IMO

Siyet · 2026-06-14T19:27:10Z

Reviewed and tested this (built on 3.12, full unit suite green, the test_module.py subinterp check passes). The multi-phase mechanics look correct. A couple of non-blocking things on top of what's already in the thread:

This overlaps PR Use PyModule_AddObjectRef over PyModule_AddObject #1059 (which converts the same PyModule_AddObject calls in _core_exec to PyModule_AddObjectRef). Whichever lands second needs a rebase, worth sequencing them deliberately.
On 3.10/3.11 the Py_mod_multiple_interpreters slot doesn't exist, so subinterpreter rejection is structurally absent there (CPython limitation, not this PR). A one-line comment near module_slots noting that would help future readers.

Built and suite-verified on 3.12.

…, m_free spacing

Siyet · 2026-06-15T21:11:32Z

Pushed the small review nits we discussed:

corrected the _core_state comment (m_clear does not touch the pointer; only m_free resets it; also dropped the overstated "guaranteed at most one instance per process" wording)
dropped the dead PyUnstable_Module_SetGIL block, the {Py_mod_gil, Py_MOD_GIL_NOT_USED} slot already covers it (per @sobolevn)
.m_free spacing

Left out of scope as agreed: the missing Py_DECREF(temp_module) cleanups (separate PR per the thread) and the global-state -> heap-types rework.

provinzkraut · 2026-06-15T21:13:21Z

Currently blocked until we have moved to heap types. See #1024 (comment)

I'll revert this to a draft

provinzkraut commented Apr 18, 2026

View reviewed changes

provinzkraut force-pushed the multi-phase-init branch 2 times, most recently from d65661e to 16fe68d Compare April 18, 2026 15:35

provinzkraut marked this pull request as draft April 18, 2026 15:36

provinzkraut marked this pull request as ready for review April 18, 2026 21:13

provinzkraut commented Apr 18, 2026

View reviewed changes

implement multi-phase init

06be1a7

provinzkraut force-pushed the multi-phase-init branch from 74741fb to 06be1a7 Compare April 18, 2026 21:18

Siyet reviewed Apr 20, 2026

View reviewed changes

Comment thread src/msgspec/_core.c Outdated

goodboy mentioned this pull request Apr 24, 2026

Port to new concurrent.interpreters support in cpython 3.14+ #1026

Open

Siyet reviewed May 26, 2026

View reviewed changes

Comment thread src/msgspec/_core.c Outdated

Comment thread tests/unit/test_module.py Outdated

Siyet reviewed May 26, 2026

View reviewed changes

Comment thread src/msgspec/_core.c Outdated

Comment thread tests/unit/test_module.py Outdated

provinzkraut added 4 commits May 31, 2026 11:29

Address review

d3f96d7

Merge branch 'main' into multi-phase-init

8203692

formatting

d37222d

do not set 'Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED' on <3.12

d396f44

provinzkraut force-pushed the multi-phase-init branch from 4a5d5a1 to d396f44 Compare May 31, 2026 10:19

sobolevn reviewed Jun 7, 2026

View reviewed changes

Merge branch 'main' into multi-phase-init

2f4b0d4

provinzkraut temporarily deployed to docs-preview June 12, 2026 13:57 — with GitHub Actions Inactive

Address review nits: correct _core_state comment, drop dead GIL block…

addc8b9

…, m_free spacing

Siyet deployed to docs-preview June 15, 2026 21:05 — with GitHub Actions View deployment

provinzkraut marked this pull request as draft June 15, 2026 21:13

	.m_free =(freefunc)msgspec_free,
	.m_free = (freefunc)msgspec_free,

Uh oh!

Conversation

provinzkraut commented Apr 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Siyet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

goodboy commented Apr 21, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Siyet commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

provinzkraut commented May 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ofek Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Siyet commented Jun 14, 2026

Uh oh!

Siyet commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

provinzkraut commented Jun 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Siyet commented May 26, 2026 •

edited

Loading

ofek Jun 13, 2026 •

edited

Loading

Siyet commented Jun 15, 2026 •

edited

Loading