test_interpreters Failing Intermittently Under Py_DISABLE_GIL

[ericsnowcurrently]

Bug report

Bug description:

On one of my recent PRs, gh-118157, I was getting consistent failures on CI in test_interpreters. Different tests would fail, but the error was always the same:

(expand)

0:08:24 load avg: 4.18 Re-running 1 failed tests in verbose mode in subprocesses
0:08:24 load avg: 4.18 Run 1 test in parallel using 1 worker process (timeout: 10 min, worker timeout: 15 min)
0:08:26 load avg: 3.93 [1/1/1] test_interpreters failed (6 failures)
Re-running test_interpreters in verbose mode (matching: test_display_preserved_exception, test_gh_109793, test_sys_path_0, test_sys_path_0, test_sys_path_0, test_sys_path_0)
test_display_preserved_exception (test.test_interpreters.test_api.TestInterpreterExec.test_display_preserved_exception) ... FAIL
test_gh_109793 (test.test_interpreters.test_lifecycle.FinalizationTests.test_gh_109793) ... FAIL
test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) ... 
### start subtest debug ###
# python3 script.py failed:
--- stdout ---
{
 "main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
 "sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000119aed0>
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x00005639aed8ea10)

------
### end subtest debug ###
 test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 script.py] ... FAIL
### start subtest debug ###
# python3 -m script failed:
--- stdout ---
{
 "main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
 "sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000117c0d0>
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x0000564591b58a10)

------
### end subtest debug ###
 test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m script] ... FAIL
### start subtest debug ###
# python3 -m pkg failed:
--- stdout ---
{
 "main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
 "sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000118c0d0>
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x000055aaa11eaa10)

------
### end subtest debug ###
 test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg] ... FAIL
### start subtest debug ###
# python3 -m pkg.script failed:
--- stdout ---
{
 "main": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4",
 "sub": "/tmp/test_python_tk7ha1xh/test_interpreters_76l2y2f4"
}
------
--- stderr ---
Exception ignored in: <function Interpreter.__del__ at 0x2000118c0d0>
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x00005637ca7cfa10)

------
### end subtest debug ###
 test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg.script] ... FAIL

======================================================================
FAIL: test_display_preserved_exception (test.test_interpreters.test_api.TestInterpreterExec.test_display_preserved_exception)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/__init__.py", line 2603, in wrapper
 return func(*args, **kwargs)
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_api.py", line 764, in test_display_preserved_exception
 self.assertEqual(stderr, dedent(f"""\
 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
 Traceback (most recent call last):
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ...<18 lines>...
 RuntimeError: uh-oh!
 ^^^^^^^^^^^^^^^^^^^^
 """))
 ^^^^^
AssertionError: 'Trac[778 chars]oh!\nException ignored in: <function Interpret[519 chars]\n\n' != 'Trac[778 chars]oh!\n'
 Traceback (most recent call last):
 File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/script.py", line 9, in <module>
 interp.exec(script)
 ~~~~~~~~~~~^^^^^^^^
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 227, in exec
 raise ExecutionFailed(excinfo)
 test.support.interpreters.ExecutionFailed: RuntimeError: uh-oh!
 
 Uncaught in the interpreter:
 
 Traceback (most recent call last):
 File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/script.py", line 6, in script
 spam.eggs()
 ~~~~~~~~~^^
 File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/spam.py", line 6, in eggs
 ham()
 ~~~^^
 File "/tmp/test_python_tk7ha1xh/tmpkqxiakqz/spam.py", line 3, in ham
 raise RuntimeError('uh-oh!')
 RuntimeError: uh-oh!
- Exception ignored in: <function Interpreter.__del__ at 0x2000119aed0>
- Traceback (most recent call last):
- File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
- File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
- TypeError: catching classes that do not inherit from BaseException is not allowed
- Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
- Python runtime state: finalizing (tstate=0x000055b06f0a1a10)
- 


======================================================================
FAIL: test_gh_10[979](https://github.com/python/cpython/actions/runs/8978599353/job/24660038142#step:19:980)3 (test.test_interpreters.test_lifecycle.FinalizationTests.test_gh_109793)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 185, in test_gh_109793
 self.assertEqual(proc.returncode, 1)
 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 1

======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 script.py]
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
 yield ctx
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
 out = self.run_python(argv, cwd=cwd)
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
 self.assertEqual(proc.returncode, 0)
 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0

======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m script]
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
 yield ctx
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
 out = self.run_python(argv, cwd=cwd)
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
 self.assertEqual(proc.returncode, 0)
 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0

======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg]
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
 yield ctx
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
 out = self.run_python(argv, cwd=cwd)
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
 self.assertEqual(proc.returncode, 0)
 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0

======================================================================
FAIL: test_sys_path_0 (test.test_interpreters.test_lifecycle.StartupTests.test_sys_path_0) [python3 -m pkg.script]
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 29, in subTest
 yield ctx
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 157, in test_sys_path_0
 out = self.run_python(argv, cwd=cwd)
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/test_interpreters/test_lifecycle.py", line 114, in run_python
 self.assertEqual(proc.returncode, 0)
 ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
AssertionError: -6 != 0

----------------------------------------------------------------------
Ran 3 tests in 1.720s

FAILED (failures=6)
test test_interpreters failed
1 test failed again:
 test_interpreters

The error looks like this:

Exception ignored in: <function Interpreter.__del__ at 0x2000117c0d0>
Traceback (most recent call last):
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 157, in __del__
 File "/home/runner/work/cpython/cpython-ro-srcdir/Lib/test/support/interpreters/__init__.py", line 173, in _decref
TypeError: catching classes that do not inherit from BaseException is not allowed
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0x0000564591b58a10)

I'm fairly sure that means one of the 3 calls in _interpreters._decref() is raising a bogus exception.

It isn't clear to me how gh-118157 would have caused this to happen. As a very temporary solution, I've skipped test_interpreters on free-threading builds.

CC @colesbury

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

[colesbury]

I think what's happening is:

• The free-threaded build is immortalizing things like modules and their module.__dict__. This is not behavior that we want long term, but it's an important performance work around for 3.13. (See Temporarily immortalize objects that use deferred reference counting #117783)
• This leads to slightly different behavior during interpreter shutdown. A lot of finalizers normally get called during finalize_remove_modules, but here that's not happening because the module's dictionary was immortalized.
• Instead, the finalizer is called slightly later in finalize_modules_clear_weaklist when we start clearing the module's __dict__ in _PyModule_Clear.

By the time Interpreter.__del__ is called, a lot of global variables including _interpreters have already been deleted from the module's dictionary so you get a bunch of weird errors, and the interpreter is not deleted.

[ericsnowcurrently]

That makes sense. You have any thoughts already on how to resolve this?

[colesbury]

I'm not sure. We could call the finalizers on all GC-enabled objects during shutdown in the free-threaded build (before we start clearing module's dictionaries). I think that makes logical sense: everything is going away, why not call their finalizers? That seems to fix test_interpreters, but a few other test cases would need tweaking.

[kumaraditya303]

On current main, test_interpreters passes under free-threading as deferred reference counting is implemented now, the tests can be enabled back now.

== CPython 3.15.0a0 free-threading build (heads/main-dirty:75e2c5dd346, Jul 14 2025, 17:20:43) [Clang 17.0.0 (clang-1700.0.13.5)]
== macOS-15.5-arm64-arm-64bit-Mach-O little-endian
== Python build: free_threading debug
== cwd: /Users/kumaraditya/work/cpython/build/test_python_worker_53278æ
== CPU count: 10
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 2517149026
0:00:00 load avg: 8.09 Run 1 test sequentially in a single process
0:00:00 load avg: 8.09 [1/1] test_interpreters
0:00:07 load avg: 7.60 [1/1] test_interpreters passed

== Tests result: SUCCESS ==

1 test OK.

Total duration: 7.1 sec
Total tests: run=165 skipped=8
Total test files: run=1/1
Result: SUCCESS