Main segfaults importing `_pyrepl` with low value for `JUMP_BACKWARD_INITIAL_VALUE`

[devdanzin]

Crash report

What happened?

Edit: I'm seeing this crash even without patching JUMP_BACKWARD_INITIAL_VALUE. Will try to figure out a MRE for unpatched crash.

If we #define JUMP_BACKWARD_INITIAL_VALUE 702 (or lower) in Include/internal/pycore_backoff.h, a debug JIT build will segfault when trying to run to the new REPL. Can also be triggered by:

python -m _pyrepl

Using some other modules like random or http.server doesn't crash.

An even lower value like 344 will segfault earlier, during the build process.

This happens since #136307 landed in 377b787.

The diff below, which is part of that commit, is enough to cause the segfault together with defining JUMP_BACKWARD_INITIAL_VALUE to 702 or lower:

diff --git a/Include/internal/pycore_global_objects_fini_generated.h b/Include/internal/pycore_global_objects_fini_generated.h
index 493377b4c25..5e7dda3a371 100644
--- a/Include/internal/pycore_global_objects_fini_generated.h
+++ b/Include/internal/pycore_global_objects_fini_generated.h
@@ -1005,6 +1005,7 @@ _PyStaticObjects_CheckRefcnt(PyInterpreterState *interp){
 _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(imag));
 _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(importlib));
 _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(in_fd));
+ _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(include_aliases));
 _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(incoming));
 _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(index));
 _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(indexgroup));
diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h
index 5dfea2f479d..6908cbf78f3 100644
--- a/Include/internal/pycore_global_strings.h
+++ b/Include/internal/pycore_global_strings.h
@@ -496,6 +496,7 @@ struct _Py_global_strings{
 STRUCT_FOR_ID(imag)
 STRUCT_FOR_ID(importlib)
 STRUCT_FOR_ID(in_fd)
+ STRUCT_FOR_ID(include_aliases)
 STRUCT_FOR_ID(incoming)
 STRUCT_FOR_ID(index)
 STRUCT_FOR_ID(indexgroup)
diff --git a/Include/internal/pycore_runtime_init_generated.h b/Include/internal/pycore_runtime_init_generated.h
index 85ced09d29d..da2ed7422c9 100644
--- a/Include/internal/pycore_runtime_init_generated.h
+++ b/Include/internal/pycore_runtime_init_generated.h
@@ -1003,6 +1003,7 @@ extern "C"{
 INIT_ID(imag), \
 INIT_ID(importlib), \
 INIT_ID(in_fd), \
+ INIT_ID(include_aliases), \
 INIT_ID(incoming), \
 INIT_ID(index), \
 INIT_ID(indexgroup), \
diff --git a/Include/internal/pycore_unicodeobject_generated.h b/Include/internal/pycore_unicodeobject_generated.h
index 6018d98d156..b1f411945e7 100644
--- a/Include/internal/pycore_unicodeobject_generated.h
+++ b/Include/internal/pycore_unicodeobject_generated.h
@@ -1772,6 +1772,10 @@ _PyUnicode_InitStaticStrings(PyInterpreterState *interp){
 _PyUnicode_InternStatic(interp, &string);
 assert(_PyUnicode_CheckConsistency(string, 1));
 assert(PyUnicode_GET_LENGTH(string) != 1);
+ string = &_Py_ID(include_aliases);
+ _PyUnicode_InternStatic(interp, &string);
+ assert(_PyUnicode_CheckConsistency(string, 1));
+ assert(PyUnicode_GET_LENGTH(string) != 1);
 string = &_Py_ID(incoming);
 _PyUnicode_InternStatic(interp, &string);
 assert(_PyUnicode_CheckConsistency(string, 1));

Backtrace:

Program received signal SIGSEGV, Segmentation fault.
_PyUnicode_Equal (str1=0x555555c88c18 <_PyRuntime+96088>, str2=0x555555c88c50 <_PyRuntime+96144>) at ./Include/object.h:815
815 return ((flags & feature) != 0);

#0 _PyUnicode_Equal (str1=0x555555c88c18 <_PyRuntime+96088>, str2=0x555555c88c50 <_PyRuntime+96144>) at ./Include/object.h:815
#1 0x00007ffff731df15 in ?? ()
#2 0x00007ffff764a751 in ?? ()
#3 0x0000555555cc2230 in _PyRuntime ()
#4 0x00007fffffffbfc0 in ?? ()
#5 0x00007ffff7317017 in ?? ()
#6 0x0000555556053f60 in ?? ()
#7 0x00007ffff7e2a9c8 in ?? ()
#8 0x00007ffff7317000 in ?? ()
#9 0x00007ffff7e2a910 in ?? ()
#10 0x0000555555cc2230 in _PyRuntime ()
#11 0x00005555557cdcf9 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x58, throwflag=1291845632)
 at Python/generated_cases.c.h:7796
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Here's the output from running PYTHON_LLTRACE=4 ./python -m _pyrepl:
lltrace_crash.txt
And here for PYTHON_OPT_DEBUG=4 ./python -m _pyrepl:
opt_debug_crash.txt

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.15.0a0 (heads/main-dirty:d7e12a362a2, Jul 29 2025, 18:24:07) [GCC 13.3.0]

Linked PRs

• GH-137218: Update make for JIT stencils #137265
• GH-137218: Fix Makefile to handle FreeBSD (fix for JIT stencil changes) #139170
• GH-137218: Fix unnecessary recompile of Programs/_freeze_module #139241

[brandtbucher]

This is only on JIT builds, correct? Curious to see the new unpatched reproducer.

[devdanzin]

This is only on JIT builds, correct?

AFAICT, after double checking, yes.

[brandtbucher]

I suspect that the bug isn't the initial value itself, but rather the code we try to compile when the value is that low. It could be that some specialization or inline cache has a nonsense value that we're not handling well, or something.

[brandtbucher]

Here's the output from running PYTHON_LLTRACE=4 ./python -m _pyrepl:

Is part of this file missing, or was the output truncated by the crash itself? The last optimized trace is incomplete (it should be length 167, but only 132 elements are printed).

Or maybe we need to add a flush call somewhere?

[devdanzin]

I believe the crash truncates it, yes.

[devdanzin]

I'm suspicious about such straightforward code segfaulting, hoping this isn't an operator error on my part.

Here's something that triggers the right crash for me, unpatched debug JIT build, please confirm whether it repros:

key = "abc"
s = 0
while True:
 key[s] == "a"

And here's something that triggers another crash, not sure what configurations are affected (will investigate tomorrow), but also crashes an unpatched debug JIT build:

def p(keys):
 s = 0
 r = []
 while s < len(keys):
 ret = ""
 while not ret and s < len(keys):
 ret = keys[s]
 s += 1
 r.extend([s])
 return r

while True:
 p("abc")

Traceback for 2nd crash:

Program received signal SIGSEGV, Segmentation fault.
0x00005555556da88a in PyObject_IsTrue (v=0x555555c88d38 <_PyRuntime+96376>) at Objects/object.c:2059
2059 else if (Py_TYPE(v)->tp_as_number != NULL &&

#0 0x00005555556da88a in PyObject_IsTrue (v=0x555555c88d38 <_PyRuntime+96376>) at Objects/object.c:2059
#1 0x00005555557d8a5b in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7e2a088, throwflag=0)
 at Python/generated_cases.c.h:11581
#2 0x00005555557db135 in _PyEval_EvalFrame (tstate=tstate@entry=0x555555cc2230 <_PyRuntime+331120>, frame=frame@entry=0x7ffff7e2a020,
 throwflag=throwflag@entry=0) at ./Include/internal/pycore_ceval.h:119
#3 0x00005555557db306 in _PyEval_Vector (tstate=tstate@entry=0x555555cc2230 <_PyRuntime+331120>, func=func@entry=0x7ffff7a4a8d0,
 locals=locals@entry=0x7ffff7a583b0, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0)
 at Python/ceval.c:1981
#4 0x00005555557db402 in PyEval_EvalCode (co=co@entry=0x7ffff785c260, globals=globals@entry=0x7ffff7a583b0,
 locals=locals@entry=0x7ffff7a583b0) at Python/ceval.c:872
#5 0x00005555558bcf17 in run_eval_code_obj (tstate=tstate@entry=0x555555cc2230 <_PyRuntime+331120>, co=co@entry=0x7ffff785c260,
 globals=globals@entry=0x7ffff7a583b0, locals=locals@entry=0x7ffff7a583b0) at Python/pythonrun.c:1365
#6 0x00005555558bd0bd in run_mod (mod=mod@entry=0x555555ee2968, filename=filename@entry=0x7ffff7aa31b0,
 globals=globals@entry=0x7ffff7a583b0, locals=locals@entry=0x7ffff7a583b0, flags=flags@entry=0x7fffffffd9b8,
 arena=arena@entry=0x7ffff7ac4760, interactive_src=0x0, generate_new_source=0) at Python/pythonrun.c:1436
#7 0x00005555558bd93d in pyrun_file (fp=fp@entry=0x555555e2e6d0, filename=filename@entry=0x7ffff7aa31b0, start=start@entry=257,
 globals=globals@entry=0x7ffff7a583b0, locals=locals@entry=0x7ffff7a583b0, closeit=closeit@entry=1, flags=0x7fffffffd9b8)
 at Python/pythonrun.c:1293
#8 0x00005555558bf1d1 in _PyRun_SimpleFileObject (fp=fp@entry=0x555555e2e6d0, filename=filename@entry=0x7ffff7aa31b0,
 closeit=closeit@entry=1, flags=flags@entry=0x7fffffffd9b8) at Python/pythonrun.c:521
#9 0x00005555558bf3c7 in _PyRun_AnyFileObject (fp=fp@entry=0x555555e2e6d0, filename=filename@entry=0x7ffff7aa31b0,
 closeit=closeit@entry=1, flags=flags@entry=0x7fffffffd9b8) at Python/pythonrun.c:81
#10 0x00005555558e872d in pymain_run_file_obj (program_name=program_name@entry=0x7ffff7aa3d80, filename=filename@entry=0x7ffff7aa31b0,
 skip_source_first_line=0) at Modules/main.c:410
#11 0x00005555558e8855 in pymain_run_file (config=config@entry=0x555555c8d2d8 <_PyRuntime+114200>) at Modules/main.c:429
#12 0x00005555558e939f in pymain_run_python (exitcode=exitcode@entry=0x7fffffffdb34) at Modules/main.c:691
#13 0x00005555558e9654 in Py_RunMain () at Modules/main.c:772

Here's the output from running with PYTHON_LLTRACE=4 for the second crash:
lltrace_2nd_crash.txt
And here's the output from PYTHON_OPT_DEBUG=4:
opt_debug_2nd_crash.txt

[brandtbucher]

Yep! This gives me a segfault:

X = "X"
while True:
 X[0] == ""

This gives me an assertion failure (_POP_TOP_UNICODE.c:121: _Py_CODEUNIT *_JIT_ENTRY(_PyInterpreterFrame *, _PyStackRef *, PyThreadState *): Assertion `PyUnicode_CheckExact(PyStackRef_AsPyObjectBorrow(value))' failed.):

X = "X"
while True:
 X[0]

Your second reproducer looks a lot like my second one here, and I wouldn't be surprised if both of these are related. Something about string indexing is broken.

[brandtbucher]

Hm, weird. After a make clean and rebuild the issue goes away.

[brandtbucher]

Maybe our make rule for the JIT is broken. I think we need to say that jit.o depends on jit_stencils*.h, not just jit_stencils.h. We now have a platform-specific header file (like jit_stencils-x86_64-pc-linux-gnu.h) and I'm not sure that the existing logic is picking that dependency up correctly.

@devdanzin, does a clean rebuild work correctly for you?

[devdanzin]

Yep! This gives me a segfault:

Great, I was really worried it was my fault 😄

@devdanzin, does a clean rebuild work correctly for you?

It still segfaults after make distclean, but if I also bypass ccache it stops crashing. Then patching and re-enabling ccache also doesn't crash, which makes no sense to me but I hope makes to you.

Edit: I simply can't repro anymore, any idea about how to bring the brokenness back?

Edit 2: BTW, I see make distclean does rm -f jit_stencils.h, should it also remove jit_stencils*.h?

[brandtbucher]

So I'm comfortable calling this a bug in the build system, not in the JIT. Are you interested in fixing it? You can ping me for help/review. Should be a couple of fairly straightforward changes in the Makefile.

CC @savannahostrowski since she might have thoughts here.

[devdanzin]

I'd rather not tackle this one, but thanks for asking.

Are you able to get back into the broken state? I wasted too much time trying to yesterday 😅.

[brandtbucher]

I think getting into the broken state is hard for the same reasons that getting out of the broken state is hard. ;)