Crash when concurrently writing with `print` and concurrently modifying `sys.stdout`

[devdanzin]

Crash report

What happened?

Playing with the code from #130148, I stumbled on a segfault in a free-threaded debug build with the following (seems to trigger faster/more consistently when pasted in the REPL):

from contextlib import redirect_stdout
from io import StringIO
from threading import Thread
import time


def test_redirect():
 text = StringIO()
 with redirect_stdout(text):
 print("hello1", file=text)
 time.sleep(0.1)
 print("hello2", file=text)
 print(text.getvalue())
 assert text.getvalue() == "hello1\nhello2\n"


for x in range(100):
 Thread(target=test_redirect, args=()).start()

I didn't have time to check whether JIT, debug or no-gil are strictly necessary for this to crash. JIT is not needed for this to crash.

Cause

The problem is that in the print() implementation, _PySys_GetAttr returns a borrowed reference.

cpython/Python/bltinmodule.c

Lines 2173 to 2175 in 655fc8a

	if (file == Py_None){
	PyThreadState *tstate = _PyThreadState_GET();
	file = _PySys_GetAttr(tstate, &_Py_ID(stdout));

So if sys.stdout changes concurrently with the print(), the program may crash because file will point to a deallocated Python object.

This affects the GIL-enabled build as well. See this reproducer: https://gist.github.com/colesbury/c48f50e95d5d68e24814a56e2664e587

Suggested fix

Introduce a _PySys_GetAttrRef that returns a new reference instead of a borrowed reference. Use that instead.

We should audit all the uses of _PySys_GetAttr and PySys_GetObject. I expect most of them will need to be replaced with functions that return new references, but it doesn't all have to be in a single PR.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a5+ experimental free-threading build (heads/main:359c7dde3bb, Feb 15 2025, 17:54:11) [GCC 11.4.0]

Linked PRs

• gh-130163: Fix usage of borrow references from _PySys_GetAttr #130282
• gh-130163: Fix possible crashes related to PySys_GetObject() #130503
• [3.13] gh-130163: Fix crashes related to PySys_GetObject() (GH-130503) #130556
• [3.13] gh-130163: Fix a leak in _pickle.c after backporting #130568
• [3.12] gh-130163: Fix crashes related to PySys_GetObject() (GH-130503) (GH-130556) #130576

[picnixz]

The issue could be with StringIO() or sys.stdout/sys.stderr or with context managers in general. I don't know where to start to investigate this =/

[tomasr8]

Here's the backtrace:

#0 PyObject_GetAttr (v=v@entry=0x2002c0f00d0, name=0x555555b83358 <_PyRuntime+110680>) at Objects/object.c:1288
#1 0x0000555555688162 in PyFile_WriteObject (v=v@entry=0x555555b836e0 <_PyRuntime+111584>, f=f@entry=0x2002c0f00d0, flags=flags@entry=1) at Objects/fileobject.c:113
#2 0x0000555555688279 in PyFile_WriteString (s=s@entry=0x5555558efa49 "\n", f=f@entry=0x2002c0f00d0) at Objects/fileobject.c:149
#3 0x000055555579c917 in builtin_print_impl (module=module@entry=0x2000025c5d0, args=args@entry=0x7fffd4bffb78, args_length=args_length@entry=1, sep=0x0, sep@entry=0x555555b410a0 <_Py_NoneStruct>, end=0x0, end@entry=0x555555b410a0 <_Py_NoneStruct>, file=0x2002c0f00d0, file@entry=0x555555b410a0 <_Py_NoneStruct>, flush=0) at Python/bltinmodule.c:2222
#4 0x000055555579ca97 in builtin_print (module=0x2000025c5d0, args=args@entry=0x7fffd4bffb78, nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Python/clinic/bltinmodule.c.h:985
#5 0x00005555556c245b in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x2000025d670, args=0x7fffd4bffb78, nargsf=<optimized out>, kwnames=0x0) at Objects/methodobject.c:452
#6 0x0000555555667519 in _PyObject_VectorcallTstate (tstate=0x555555cb4900, callable=callable@entry=0x2000025d670, args=args@entry=0x7fffd4bffb78, nargsf=9223372036854775809, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:167
#7 0x00005555556675f5 in PyObject_Vectorcall (callable=callable@entry=0x2000025d670, args=args@entry=0x7fffd4bffb78, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327
#8 0x00005555557abdd3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7e3d1b0, throwflag=<optimized out>) at Python/generated_cases.c.h:1371
#9 0x00005555557d0ca5 in _PyEval_EvalFrame (tstate=tstate@entry=0x555555cb4900, frame=frame@entry=0x7ffff7e3d020, throwflag=throwflag@entry=0) at ./Include/internal/pycore_ceval.h:116
#10 0x00005555557d0e76 in _PyEval_Vector (tstate=0x555555cb4900, func=0x2000068dc90, locals=locals@entry=0x0, args=0x7fffd4bffd48, argcount=<optimized out>, kwnames=0x0) at Python/ceval.c:1745
#11 0x0000555555667192 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:413
#12 0x000055555566a435 in _PyObject_VectorcallTstate (tstate=tstate@entry=0x555555cb4900, callable=callable@entry=0x2000068dc90, args=args@entry=0x7fffd4bffd48, nargsf=nargsf@entry=1, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:167
#13 0x000055555566a69a in method_vectorcall (method=<optimized out>, args=0x555555b87c98 <_PyRuntime+129432>, nargsf=0, kwnames=0x0) at Objects/classobject.c:72
#14 0x0000555555668e22 in _PyVectorcall_Call (tstate=tstate@entry=0x555555cb4900, func=0x55555566a4bd <method_vectorcall>, callable=callable@entry=0x200006b8170, tuple=tuple@entry=0x555555b87c70 <_PyRuntime+129392>, kwargs=kwargs@entry=0x0) at Objects/call.c:273
#15 0x0000555555669151 in _PyObject_Call (tstate=0x555555cb4900, callable=0x200006b8170, args=0x555555b87c70 <_PyRuntime+129392>, kwargs=0x0) at Objects/call.c:348
#16 0x000055555566918d in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373
#17 0x00005555558cf626 in thread_run (boot_raw=boot_raw@entry=0x555555cb48b0) at ./Modules/_threadmodule.c:354
#18 0x000055555585b7f3 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:242
#19 0x00007ffff7c9caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#20 0x00007ffff7d29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

The immediate cause seems to be related to the printing itself, but I didn't dig into it too much

[devdanzin]

The issue could be with StringIO() or sys.stdout/sys.stderr or with context managers in general.

Substituting StringIO with the following stops crashing, would it point to it being the culprit?

class FakeStringIO:
 def write(self, value):
 pass
 def getvalue(self):
 return "hello1\nhello2\n"
 def flush(self):
 pass

[picnixz]

I suspect it has something to do with the PyFile_* interface.

[tomasr8]

Might be something to do with redirect_stdout, I'm getting a segfault even with this simplified reproducer:

from contextlib import redirect_stdout
from io import StringIO
from threading import Thread
import time


def test_redirect():
 text = StringIO()
 with redirect_stdout(text):
 time.sleep(0.1)
 print(text.getvalue())


for x in range(100):
 Thread(target=test_redirect, args=()).start()

edit:

Both crashes seem to be caused by Py_TYPE returning NULL. Here's the backtrace for this repro:

#0 _Py_Dealloc (op=op@entry=0x2009c1000d0) at Objects/object.c:2994
#1 0x00005555556c8407 in _Py_DecRefSharedDebug (o=o@entry=0x2009c1000d0, filename=filename@entry=0x5555558e4237 "./Include/refcount.h", lineno=lineno@entry=502) at Objects/object.c:418
#2 0x00005555556c2569 in Py_DECREF (filename=filename@entry=0x5555558e4237 "./Include/refcount.h", lineno=lineno@entry=502, op=0x2009c1000d0) at ./Include/refcount.h:347
#3 0x00005555556c28bb in Py_XDECREF (op=<optimized out>) at ./Include/refcount.h:502
#4 0x00005555556c2ff1 in meth_dealloc (self=0x2009e0e0160) at Objects/methodobject.c:172
#5 0x00005555556c82fe in _Py_Dealloc (op=0x2009e0e0160) at Objects/object.c:3014
#6 0x00005555556c83d7 in _Py_MergeZeroLocalRefcount (op=op@entry=0x2009e0e0160) at Objects/object.c:436
#7 0x000055555568767f in Py_DECREF (filename=filename@entry=0x5555558eb2c2 "Objects/fileobject.c", lineno=lineno@entry=127, op=op@entry=0x2009e0e0160) at ./Include/refcount.h:343
#8 0x00005555556881c1 in PyFile_WriteObject (v=v@entry=0x555555b836e0 <_PyRuntime+111584>, f=f@entry=0x2009c1000d0, flags=flags@entry=1) at Objects/fileobject.c:127
#9 0x0000555555688279 in PyFile_WriteString (s=s@entry=0x5555558efa49 "\n", f=f@entry=0x2009c1000d0) at Objects/fileobject.c:149
#10 0x000055555579c917 in builtin_print_impl (module=module@entry=0x2000025c5d0, args=args@entry=0x7fff5f3ffb78, args_length=args_length@entry=1, sep=0x0, sep@entry=0x555555b410a0 <_Py_NoneStruct>, end=0x0, end@entry=0x555555b410a0 <_Py_NoneStruct>, file=0x2009c1000d0, file@entry=0x555555b410a0 <_Py_NoneStruct>, flush=0) at Python/bltinmodule.c:2222
#11 0x000055555579ca97 in builtin_print (module=0x2000025c5d0, args=args@entry=0x7fff5f3ffb78, nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Python/clinic/bltinmodule.c.h:985
#12 0x00005555556c245b in cfunction_vectorcall_FASTCALL_KEYWORDS (func=0x2000025d670, args=0x7fff5f3ffb78, nargsf=<optimized out>, kwnames=0x0) at Objects/methodobject.c:452
#13 0x0000555555667519 in _PyObject_VectorcallTstate (tstate=0x555555daaa30, callable=callable@entry=0x2000025d670, args=args@entry=0x7fff5f3ffb78, nargsf=9223372036854775809, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:167
#14 0x00005555556675f5 in PyObject_Vectorcall (callable=callable@entry=0x2000025d670, args=args@entry=0x7fff5f3ffb78, nargsf=<optimized out>, kwnames=kwnames@entry=0x0) at Objects/call.c:327
#15 0x00005555557abdd3 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff75641b0, throwflag=<optimized out>) at Python/generated_cases.c.h:1371
#16 0x00005555557d0ca5 in _PyEval_EvalFrame (tstate=tstate@entry=0x555555daaa30, frame=frame@entry=0x7ffff7564020, throwflag=throwflag@entry=0) at ./Include/internal/pycore_ceval.h:116
#17 0x00005555557d0e76 in _PyEval_Vector (tstate=0x555555daaa30, func=0x2000068dc90, locals=locals@entry=0x0, args=0x7fff5f3ffd48, argcount=<optimized out>, kwnames=0x0) at Python/ceval.c:1745
#18 0x0000555555667192 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:413
#19 0x000055555566a435 in _PyObject_VectorcallTstate (tstate=tstate@entry=0x555555daaa30, callable=callable@entry=0x2000068dc90, args=args@entry=0x7fff5f3ffd48, nargsf=nargsf@entry=1, kwnames=kwnames@entry=0x0) at ./Include/internal/pycore_call.h:167
#20 0x000055555566a69a in method_vectorcall (method=<optimized out>, args=0x555555b87c98 <_PyRuntime+129432>, nargsf=0, kwnames=0x0) at Objects/classobject.c:72
#21 0x0000555555668e22 in _PyVectorcall_Call (tstate=tstate@entry=0x555555daaa30, func=0x55555566a4bd <method_vectorcall>, callable=callable@entry=0x200006babd0, tuple=tuple@entry=0x555555b87c70 <_PyRuntime+129392>, kwargs=kwargs@entry=0x0) at Objects/call.c:273
#22 0x0000555555669151 in _PyObject_Call (tstate=0x555555daaa30, callable=0x200006babd0, args=0x555555b87c70 <_PyRuntime+129392>, kwargs=0x0) at Objects/call.c:348
#23 0x000055555566918d in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373
#24 0x00005555558cf626 in thread_run (boot_raw=boot_raw@entry=0x555555daa9e0) at ./Modules/_threadmodule.c:354
#25 0x000055555585b7f3 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:242
#26 0x00007ffff7c9caa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#27 0x00007ffff7d29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

[picnixz]

Might be something to do with redirect_stdout, I'm getting a segfault even with this simplified reproducer:

Have you tried removing the usage to StringIO()? because it appears that we crash in Py_DECREF of a file object.

[sergey-miryanov]

Yes, it is caused by redirect_stdout.

1. At some point, print from T1 and T2 uses StringIO, which is owned by T1.
2. PyObject_GetAttr (

   cpython/Objects/fileobject.c

   Line 113 in 798f8d3

   writer = PyObject_GetAttr(f, &_Py_ID(write));

   ) increments shared ref_cnt (by T2) and local ref_cnt (by T1).
3. Py_DECREF (

   cpython/Objects/fileobject.c

   Line 127 in 798f8d3

   Py_DECREF(writer);

   ) from T2 decrements shared ref_cnt and got 0 and do _Py_Dealloc
4. Py_DECREF from T1 decrements local ref_cnt, got 0 and go to _Py_MergeZeroLocalRefcount, see that shared ref_cnt is 0 and go to fast path and do _Py_Dealloc.
5. This is double free and so segfault.

The easiest way to fix this is probably to protect PyFile_WriteObject with a critical section. However, this might slightly affect the performance of single-threaded apps.

The best solution might be to update the documentation and point out this potential issue with redirect_*.