Fix segmentation fault on Python 3.13 by shadchin · Pull Request #960 · msgspec/msgspec

shadchin · 2025-11-27T11:55:09Z

Fix for #910 and #868

The problem is only with Python 3.13, in Python 3.14 PyObject_GC_New does this initialization itself, see python/cpython#123192

provinzkraut

Seems like a reasonable fix to me.

I'm curious though how you tracked it down to this?

Also, can you re-enable the regular tests for 3.13 again here?

https://github.com/jcrist/msgspec/blob/1bc64dd8fffbad221abb02aeac1e9060c239c692/.github/workflows/ci.yml#L63-L64

shadchin · 2025-11-27T13:01:35Z

I'm curious though how you tracked it down to this?

It was difficult :) The short version:

From gdb bt found following calls:

4  make_dict_from_instance_attributes
#5  _PyObject_MaterializeManagedDict_LockHeld
#6  _PyObject_MaterializeManagedDict
#7  ensure_managed_dict

In next step, I found that values was not initialized:

(gdb) frame 4
#4  0x00000000023d7c59 in make_dict_from_instance_attributes (interp=0x2fcc2e8 <_PyRuntime+104400>, keys=0x7ffff4e24f20, values=0x7ffff54dc070) at /home/shadchin/arc/arcadia/contrib/tools/python3/Objects/dictobject.c:6736
6736                track += _PyObject_GC_MAY_BE_TRACKED(val);
(gdb) info locals
val = <unknown at remote 0xcdcdcdcdcdcdcdcd>
i = 0
used = 1
track = 0
size = 30
res = 0x7ffff1b535b0
(gdb) print values
$1 = (PyDictValues *) 0x7ffff54dc070
(gdb) print keys
$2 = (PyDictKeysObject *) 0x7ffff4e24f20
(gdb) x/10gx values
0x7ffff54dc070: 0xcdcdcdcdcdcdcdcd      0xcdcdcdcdcdcdcdcd
0x7ffff54dc080: 0xcdcdcdcdcdcdcdcd      0xcdcdcdcdcdcdcdcd
0x7ffff54dc090: 0xcdcdcdcdcdcdcdcd      0xcdcdcdcdcdcdcdcd
0x7ffff54dc0a0: 0xcdcdcdcdcdcdcdcd      0xcdcdcdcdcdcdcdcd
0x7ffff54dc0b0: 0xcdcdcdcdcdcdcdcd      0xcdcdcdcdcdcdcdcd

Then I had to study what managed dict/PyDictValues and Struct_alloc. I found that it's true for 3.13 that we don't initialize PyDictValues, looked at the CPython code, how it does it, found the _PyObject_InitInlineValues call.

ofek · 2025-11-27T15:42:38Z

Do we know why 3.14 is fine?

provinzkraut · 2025-11-27T15:55:49Z

@shadchin seems like _PyObject_InitInlineValues isn't available?

https://github.com/jcrist/msgspec/actions/runs/19736064121/job/56566395047?pr=960#step:8:228

shadchin · 2025-11-27T17:31:47Z

Do we know why 3.14 is fine?

Python 3.14+ does this initialization itself - https://github.com/python/cpython/blob/main/Python/gc.c#L2377-L2379

provinzkraut · 2025-11-27T18:01:20Z

    memset((char *)obj + sizeof(PyObject), '\0', type->tp_basicsize - sizeof(PyObject));
+#if PY313_PLUS && !PY314_PLUS
+    if (type->tp_flags & Py_TPFLAGS_INLINE_VALUES) {
+        _PyObject_InitInlineValues(obj, type);


Maybe it's worth asking upstream about how to deal with this without relying on private APIs?

Not like we don't rely on private APIs elsewhere, so I wouldn't consider it a blocker, but if there's a way, that might be nice?

shadchin · 2025-11-27T18:51:26Z

@shadchin seems like _PyObject_InitInlineValues isn't available?

https://github.com/jcrist/msgspec/actions/runs/19736064121/job/56566395047?pr=960#step:8:228

We have our own build system, fix is working for us, but I forgot to check on a clean environment, sorry. I tried to make a fix but it didn't help. There's a conflict with write_u64.

I'll go upstream with it.

shadchin · 2025-12-13T16:45:17Z

I took a different approach and it helped)

Maybe something else will prompt in upstream.

ZeroIntensity

Coming from upstream

ZeroIntensity · 2025-12-16T13:51:57Z

+#endif
+
    if (is_gc) {
        obj = PyObject_GC_New(PyObject, type);


Manually calling PyObject_GC_New/PyObject_New doesn't really work for subtypes. It might work in practice sometimes, but it's not something that's supposed to be supported.

I believe the root of this bug comes from the fact that on 3.13, inline values are initialized in the tp_alloc slot of type objects, which is skipped here. In 3.14, we moved that logic to PyObject_GC_New because we decided to support more types.

So, to fix this, just call type->tp_alloc instead; that will invoke the supertype (PyType_GenericAlloc) and initialize the inline values.

Thanks, I switched to type->tp_alloc

ZeroIntensity · 2025-12-16T13:53:16Z

    bool is_gc = MS_TYPE_IS_GC(type);

+#if PY313_PLUS && !PY314_PLUS
+    type->tp_flags &= ~Py_TPFLAGS_INLINE_VALUES;


I'm almost certain you'll trigger assertion failures (or worse, get spurious crashes) with this. Supertypes don't expect their flags to be different in subtypes.

ofek · 2025-12-17T07:10:52Z

Thanks! How would that change affect performance?

shadchin · 2025-12-19T07:38:54Z

Thanks! How would that change affect performance?

https://github.com/jcrist/msgspec/blob/main/benchmarks/bench_structs.py

main:

+----------------------+-------------+-------------+---------------+------------+
|                      | import (μs) | create (μs) | equality (μs) | order (μs) |
+======================+=============+=============+===============+============+
| **msgspec**          | 12.82       | 0.10        | 0.02          | 0.03       |
+----------------------+-------------+-------------+---------------+------------+
| **standard classes** | 9.68        | 0.43        | 0.07          | 0.21       |
+----------------------+-------------+-------------+---------------+------------+
| **attrs**            | 466.53      | 0.43        | 0.07          | 2.10       |
+----------------------+-------------+-------------+---------------+------------+
| **dataclasses**      | 606.02      | 0.44        | 0.19          | 0.21       |
+----------------------+-------------+-------------+---------------+------------+
| **pydantic**         | 320.21      | 1.84        | 1.01          | N/A        |
+----------------------+-------------+-------------+---------------+------------+

this pr:

+----------------------+-------------+-------------+---------------+------------+
|                      | import (μs) | create (μs) | equality (μs) | order (μs) |
+======================+=============+=============+===============+============+
| **msgspec**          | 13.75       | 0.11        | 0.02          | 0.03       |
+----------------------+-------------+-------------+---------------+------------+
| **standard classes** | 10.17       | 0.46        | 0.08          | 0.22       |
+----------------------+-------------+-------------+---------------+------------+
| **attrs**            | 497.77      | 0.48        | 0.07          | 2.24       |
+----------------------+-------------+-------------+---------------+------------+
| **dataclasses**      | 651.33      | 0.48        | 0.20          | 0.21       |
+----------------------+-------------+-------------+---------------+------------+
| **pydantic**         | 340.21      | 1.86        | 1.08          | N/A        |
+----------------------+-------------+-------------+---------------+------------+

https://github.com/jcrist/msgspec/blob/main/benchmarks/bench_gc.py

main:

+-----------------------------------+--------------+-------------------+
|                                   | GC time (ms) | Memory Used (MiB) |
+===================================+==============+===================+
| **standard class**                | 133.60       | 204.03            |
+-----------------------------------+--------------+-------------------+
| **standard class with __slots__** | 96.68        | 120.11            |
+-----------------------------------+--------------+-------------------+
| **msgspec struct**                | 24.18        | 120.11            |
+-----------------------------------+--------------+-------------------+
| **msgspec struct with gc=False**  | 2.38         | 104.85            |
+-----------------------------------+--------------+-------------------+

this pr:

+-----------------------------------+--------------+-------------------+
|                                   | GC time (ms) | Memory Used (MiB) |
+===================================+==============+===================+
| **standard class**                | 132.79       | 204.03            |
+-----------------------------------+--------------+-------------------+
| **standard class with __slots__** | 97.56        | 120.11            |
+-----------------------------------+--------------+-------------------+
| **msgspec struct**                | 23.94        | 120.11            |
+-----------------------------------+--------------+-------------------+
| **msgspec struct with gc=False**  | 2.39         | 104.85            |
+-----------------------------------+--------------+-------------------+

python -VV
Python 3.12.11 (main, Jun  4 2025, 08:56:18) [GCC 9.4.0]

shadchin · 2026-01-04T16:40:17Z

gentle ping)

shadchin · 2026-01-19T08:25:37Z

Can you tell me what the next course of action is? Do you need any more help from me?

westernmagic · 2026-02-02T15:11:57Z

As mentioned in #868, I can confirm this solves that issue

alldefector · 2026-03-16T05:38:07Z

We're hitting a related segfault on Python 3.14 (msgspec 0.20.0) — filed as #989.

The crash is in json_decode at a consistent offset, with segfault at 20 (NULL + 0x20 dereference). We believe Struct_alloc's memset is clobbering the inline values initialization that PyObject_GC_New performs on 3.14. Your fix (using tp_alloc instead) would resolve this as well.

While the original issue (#910, #868) was reported as 3.13-only, the same Struct_alloc pattern also appears unsafe on 3.14 — PyObject_GC_New initializes inline values, but memset immediately zeroes them out.

This is affecting us in production (FastAPI + gunicorn, concurrent JSON decoding). Would love to see this merged. Happy to test the branch if that helps move things forward.

Siyet · 2026-04-08T12:11:02Z

Ported to the community fork msgspec-arise: PR #28

jcrist

Thanks! LGTM!

jcrist · 2026-04-08T18:08:36Z


-/* Note this always allocates an UNTRACKED object */
 static PyObject *
 Struct_alloc(PyTypeObject *type) {


The hackiness here was originally to allocate structs with gc=False as untracked objects. This meant that workflows allocating a lot of them (say decoding a large message) wouldn't increment the GC counter at all, and could avoid unnecessary GC cycles. However, this required tons of hackery and cpython internals mucking. I think at this point relying on the standard upstream methods (tp_alloc) is worth it. The optimization is less effective, but still:

Slightly reduces memory usage per struct object

Reduces the overall number of objects a GC pass needs to traverse

Fix segmentation fault on Python 3.13

452fddb

shadchin mentioned this pull request Nov 27, 2025

Segmentation fault on Python 3.13 when running under ASan #910

Closed

provinzkraut approved these changes Nov 27, 2025

View reviewed changes

Re-enable the regular tests for 3.13

d14b921

Add include

639f7ee

provinzkraut reviewed Nov 27, 2025

View reviewed changes

shadchin mentioned this pull request Dec 13, 2025

Crash msgspec with Python 3.13.* python/cpython#142671

Closed

Disable Py_TPFLAGS_INLINE_VALUES

6ffe4e3

ZeroIntensity reviewed Dec 16, 2025

View reviewed changes

Switch to type->tp_alloc

24da222

oprypin approved these changes Feb 25, 2026

View reviewed changes

alldefector mentioned this pull request Mar 16, 2026

Segmentation fault in json_decode on Python 3.14 — Struct_alloc memset clobbers inline values #989

Closed

Merge branch 'main' into patch-1

225dcd5

jcrist approved these changes Apr 8, 2026

View reviewed changes

jcrist merged commit a5a5ca2 into msgspec:main Apr 8, 2026
22 checks passed

shadchin deleted the patch-1 branch April 8, 2026 18:50

Uh oh!

Conversation

shadchin commented Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

provinzkraut left a comment

Choose a reason for hiding this comment

Uh oh!

shadchin commented Nov 27, 2025

Uh oh!

ofek commented Nov 27, 2025

Uh oh!

provinzkraut commented Nov 27, 2025

Uh oh!

shadchin commented Nov 27, 2025

Uh oh!

provinzkraut Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

shadchin commented Nov 27, 2025

Uh oh!

shadchin commented Dec 13, 2025

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

ZeroIntensity Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

shadchin Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

ZeroIntensity Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

ofek commented Dec 17, 2025

Uh oh!

shadchin commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shadchin commented Jan 4, 2026

Uh oh!

shadchin commented Jan 19, 2026

Uh oh!

westernmagic commented Feb 2, 2026

Uh oh!

alldefector commented Mar 16, 2026

Uh oh!

Siyet commented Apr 8, 2026

Uh oh!

jcrist left a comment

Choose a reason for hiding this comment

Uh oh!

jcrist Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

shadchin commented Nov 27, 2025 •

edited

Loading

shadchin commented Dec 19, 2025 •

edited

Loading