Regression in Python 3.14.0a7.

[felixxm]

Crash report

What happened?

I noticed a regression in Python 3.14.0 alpha 7. One of tests from Django test suite crashes test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test:

$ ./runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
Fatal Python error: Segmentation fault

Current thread 0x00007858cbcc2b80 [python] (most recent call first):
 File "/django/django/test/runner.py", line 185 in _confirm_picklable
 File "/django/django/test/runner.py", line 215 in check_picklable
 File "/django/django/test/runner.py", line 294 in addError
 File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 246 in _addClassOrModuleLevelException
 File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 236 in _createClassOrModuleLevelException
 File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 176 in _handleClassSetUp
 File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 114 in run
 File "/django/tests/test_runner/test_parallel.py", line 225 in test_handle_add_error_before_first_test
 File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/case.py", line 615 in _callTestMethod
 File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/case.py", line 669 in run
 File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/case.py", line 725 in __call__
 File "/django/django/test/testcases.py", line 357 in _setup_and_call
 File "/django/django/test/testcases.py", line 302 in __call__
 File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 122 in run
 File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 84 in __call__
 File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/runner.py", line 259 in run
 File "/django/django/test/runner.py", line 1020 in run_suite
 File "/django/django/test/runner.py", line 1093 in run_tests
 File "/django/tests/./runtests.py", line 427 in django_tests
 File "/django/tests/./runtests.py", line 788 in <module>

Current thread's C stack trace (most recent call first):
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _Py_DumpStack+0x45 [0x7858cb8e3145]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2f85d4 [0x7858cb8f85d4]
 Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x42520 [0x7858cb242520]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at PyObject_GC_UnTrack+0x19 [0x7858cb887269]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2e08a3 [0x7858cb8e08a3]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x10cb15 [0x7858cb70cb15]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x10d235 [0x7858cb70d235]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x18e862 [0x7858cb78e862]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x8ec9 [0x7858cb699059]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfea03 [0x7858cb6fea03]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1e48 [0x7858cb691fd8]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfc8ea [0x7858cb6fc8ea]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfcb40 [0x7858cb6fcb40]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x19d144 [0x7858cb79d144]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyObject_MakeTpCall+0x86 [0x7858cb6fa506]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1c1e [0x7858cb691dae]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfea03 [0x7858cb6fea03]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1e48 [0x7858cb691fd8]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfc8ea [0x7858cb6fc8ea]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfcb40 [0x7858cb6fcb40]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x19d144 [0x7858cb79d144]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyObject_MakeTpCall+0x86 [0x7858cb6fa506]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1c1e [0x7858cb691dae]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at PyEval_EvalCode+0x1dd [0x7858cb854ced]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2c57ad [0x7858cb8c57ad]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2c7996 [0x7858cb8c7996]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2c7f3f [0x7858cb8c7f3f]
 Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at Py_RunMain+0x996 [0x7858cb8f64e6]
 <truncated rest of calls>

Extension modules: pywatchman.bser, markupsafe._speedups (total: 2)
Segmentation fault (core dumped)

It also crashes on the current main branch of cpython (reproduced at 1550c30 compiled with debug):

$ runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
./Include/refcount.h:526: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 0x7fc6598b3930 is freed>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized
Traceback (most recent call last):
 File "/cpython/Lib/unittest/suite.py", line 166, in _handleClassSetUp
 setUpClass()
 ~~~~~~~~~~^^
 File "/django/tests/test_runner/test_parallel.py", line 74, in setUpClass
 raise ValueError("woops")
ValueError: woops

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
 File "/django/tests/.test-python-dev/lib/python3.14/site-packages/tblib/pickling_support.py", line 30, in unpickle_exception
 inst.__traceback__ = tb
 ^^^^^^^^^^^^^^^^^^
TypeError: __traceback__ must be a traceback or None

Extension modules: pywatchman.bser, markupsafe._speedups (total: 2)
Aborted (core dumped)

I bisected this regression to the 053c285. Previously it passed with an assertion warning

./runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/workspace/repo/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.002s

OK
Objects/typeobject.c:6226: _Py_NegativeRefcount: Assertion failed: object has negative ref count
Enable tracemalloc to get the memory block allocation traceback

object address : 0x7cd1aa2aa1c0
object refcount : 4294967295
object type : 0x63d9c0393140
object type name: type
object repr : <class 'bserobj_tuple'>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x000063d9c0404558)

Current thread 0x00007cd1ab58b740 (most recent call first):
 <no Python frame>
Aborted (core dumped)

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a7+ (heads/main:6a9bfee4fa1, Apr 25 2025, 10:31:59) [GCC 11.4.0]

[StanFromIreland]

Can you please link the test code?

[felixxm]

Can you please link the test code?

Added link to the issue description.

[ZeroIntensity]

Looking at the faulthandler output, it looks like there are two extensions active, so it's possible this isn't our fault. (There have been some recent reference counting changes; if someone was incorrectly relying on some implementation details, then crashing makes sense.)

Would you mind isolating this down to something using the C API, or at least some Python code outside of a test suite?

[felixxm]

Looking at the faulthandler output, it looks like there are two extensions active, so it's possible this isn't our fault. (There have been some recent reference counting changes; if someone was incorrectly relying on some implementation details, then crashing makes sense.)

I uninstalled both packages (pywatchman and markupsafe) but the crash still occurs.

Would you mind isolating this down to something using the C API, or at least some Python code outside of a test suite?

Will try 🕵

[sobolevn]

cc @mpage

[mpage]

Will take a look on monday

[mpage]

I think the root cause of this issue was fixed by gh-133383. I cannot reproduce the crash on main at f5b7847. I can reproduce the crash if I revert 78adb63 on top of f5b7847.

[felixxm]

I think the root cause of this issue was fixed by gh-133383. I cannot reproduce the crash on main at f5b7847. I can reproduce the crash if I revert 78adb63 on top of f5b7847.

You're right, thanks 👍 It still reports Objects/typeobject.c:6216: _Py_NegativeRefcount: Assertion failed: object has negative ref count, but no longer crashes.

[mpage]

Hmm ... I'm not able to reproduce the assertion failure on main (or alpha 6). I see output that looks like:

(django_a6) > ./runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/data/users/mpage/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.003s

OK
(django_a6) > python -VV
Python 3.14.0a6 (tags/v3.14.0a6:77b2c933cab, May 5 2025, 14:12:02) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)]
(django_a6) >

Are you able to bisect the assertion failure?

[felixxm]

Hmm ... I'm not able to reproduce the assertion failure on main (or alpha 6). I see output that looks like:

Maybe because I configure python with --with-pydebug 🤔

./configure --with-pydebug && make -j

Are you able to bisect the assertion failure?

Should be doable, give me 1-2 days.

[mpage]

Maybe because I configure python with --with-pydebug

Hrm, I'm also compiling with --with-pydebug...

[ilovelinux]

I compiled Python from 3a64844 (main ~ 3.15) and 03457ca (3.14 branch) and I can't replicate either.

I ran tests in a Python venv after installing django/django@6aa05fd