gh-140260: Fix data race in _struct module initialization with subinterpreters#140909

ashm-dev · 2025-11-02T12:29:49Z

Issue: ThreadSanitizer: data race in _struct module initialization with InterpreterPoolExecutor (free-threading build) #140260

Modules/_struct.c

ZeroIntensity

Please add a test stressing this.

Modules/_struct.c

Lib/test/test_struct.py

Modules/_struct.c

Lib/test/test_struct.py

Tools/c-analyzer/cpython/globals-to-fix.tsv

ZeroIntensity · 2025-11-05T13:46:37Z

I think the failing job will be solved by a rebase.

ZeroIntensity

LGTM

ZeroIntensity · 2025-11-05T13:52:35Z

Hm, that failure looks unrelated. @encukou, is it possible that #138190 broke something?

encukou · 2025-11-06T06:59:22Z

It did (now fixed thanks to Victor), but I don't think it broke mobile. Maybe it's sequential tests, or cross-compilation?

kumaraditya303 · 2025-11-11T20:23:38Z

Seems like Android and iOS do not support subinterpreters so the test should be skipped on those platforms.

ZeroIntensity · 2025-11-11T20:32:15Z

Good catch, I disabled auto-merge for the time being.

miss-islington-app · 2025-11-13T11:01:35Z

Thanks @ashm-dev for the PR, and @kumaraditya303 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

…h subinterpreters (pythonGH-140909) (cherry picked from commit 63548b3) Co-authored-by: Shamil <[email protected]>

bedevere-app · 2025-11-13T11:01:44Z

GH-141501 is a backport of this pull request to the 3.14 branch.

bedevere-bot · 2025-11-13T11:23:38Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian root 3.x (tier-1) has failed when building commit 63548b3.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/345/builds/12669) and take a look at the build logs.
Check if the failure is related to this commit (63548b3) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/345/builds/12669

Failed tests:

test.test_os.test_os

Failed subtests:

test_timerfd_interval - test.test_os.test_os.TimerfdTests.test_timerfd_interval

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last): File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_os/test_os.py", line 4017, in test_timerfd_intervalself.assertEqual(self.read_count_signaled(fd), count) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^AssertionError: 4 != 3

…th subinterpreters (GH-140909) (#141501) gh-140260: fix data race in `_struct` module initialization with subinterpreters (GH-140909) (cherry picked from commit 63548b3) Co-authored-by: Shamil <[email protected]>

* main: (463 commits) pythongh-140601: Add ResourceWarning to iterparse when not closed (pythonGH-140603) pythongh-137969: Fix double evaluation of `ForwardRef`s which rely on globals (python#140974) pythongh-139109: A new tracing JIT compiler frontend for CPython (pythonGH-140310) pythongh-141004: Document `PyErr_RangedSyntaxLocationObject` (python#141521) pythongh-140873: Add support of non-descriptor callables in functools.singledispatchmethod() (pythonGH-140884) pythongh-139653: Add PyUnstable_ThreadState_SetStackProtection() (python#139668) pythongh-141004: Document `PyCode_Optimize` (pythonGH-141378) pythongh-141004: Document C APIs for dictionary keys, values, and items (pythonGH-141009) pythongh-137959: Fix `TIER1_TO_TIER2` macro name in JIT InternalDocs (pythonGH-141496) pythongh-139871: Add `bytearray.take_bytes([n])` to efficiently extract `bytes` (pythonGH-140128) pythongh-140601: Refactor ElementTree.iterparse() tests (pythonGH-141499) pythongh-135801: Add the module parameter to compile() etc (pythonGH-139652) pythongh-140260: fix data race in `_struct` module initialization with subinterpreters (python#140909) pythongh-137109: refactor warning about threads when forking (python#141438) pythongh-141004: Document `PyRun_InteractiveOneObject` (pythonGH-141405) pythongh-124111: Fix TCL 9 thread detection (pythonGH-128103) pythongh-141442: Add escaping to iOS testbed arguments (python#141443) pythongh-140936: Fix JIT assertion crash at finalization if some generator is alive (pythonGH-140969) Add details about JIT build infrastructure and updating dependencies to `Tools/jit` (python#141167) pythongh-141412: Use reliable target URL for urllib example (pythonGH-141428) ...

* 'main' of github.com:python/cpython: (464 commits) pythongh-140601: Add ResourceWarning to iterparse when not closed (pythonGH-140603) pythongh-137969: Fix double evaluation of `ForwardRef`s which rely on globals (python#140974) pythongh-139109: A new tracing JIT compiler frontend for CPython (pythonGH-140310) pythongh-141004: Document `PyErr_RangedSyntaxLocationObject` (python#141521) pythongh-140873: Add support of non-descriptor callables in functools.singledispatchmethod() (pythonGH-140884) pythongh-139653: Add PyUnstable_ThreadState_SetStackProtection() (python#139668) pythongh-141004: Document `PyCode_Optimize` (pythonGH-141378) pythongh-141004: Document C APIs for dictionary keys, values, and items (pythonGH-141009) pythongh-137959: Fix `TIER1_TO_TIER2` macro name in JIT InternalDocs (pythonGH-141496) pythongh-139871: Add `bytearray.take_bytes([n])` to efficiently extract `bytes` (pythonGH-140128) pythongh-140601: Refactor ElementTree.iterparse() tests (pythonGH-141499) pythongh-135801: Add the module parameter to compile() etc (pythonGH-139652) pythongh-140260: fix data race in `_struct` module initialization with subinterpreters (python#140909) pythongh-137109: refactor warning about threads when forking (python#141438) pythongh-141004: Document `PyRun_InteractiveOneObject` (pythonGH-141405) pythongh-124111: Fix TCL 9 thread detection (pythonGH-128103) pythongh-141442: Add escaping to iOS testbed arguments (python#141443) pythongh-140936: Fix JIT assertion crash at finalization if some generator is alive (pythonGH-140969) Add details about JIT build infrastructure and updating dependencies to `Tools/jit` (python#141167) pythongh-141412: Use reliable target URL for urllib example (pythonGH-141428) ...

mhsmith · 2025-11-20T08:28:45Z

Seems like Android and iOS do not support subinterpreters so the test should be skipped on those platforms.

Based on discussions on Discord, it looks like this failure would happen on any platform if you run the tests with --single-process. So Android and iOS actually do support subintepreters, it's just that they're the only platforms that use --single-process in GitHub Actions.

…h subinterpreters (python#140909)

Fix data race in _struct module endian table initialization
243b5f0

bedevere-appbot added the awaiting review label Nov 2, 2025

bedevere-appbot mentioned this pull request Nov 2, 2025
ThreadSanitizer: data race in _struct module initialization with InterpreterPoolExecutor (free-threading build) #140260
Closed

fix doc
4c9892e

ashm-dev requested a review from ericsnowcurrently as a code owner November 2, 2025 12:58

refactor(c-analyzer): add endian_tables_init_once to globals-to-fix list
95958e8

ashm-dev force-pushed the tsan branch from 64f49c0 to 95958e8Compare November 2, 2025 13:10

ZeroIntensity reviewed Nov 2, 2025
View reviewed changes

Modules/_struct.c Outdated Show resolvedHide resolved

ZeroIntensity added needs backport to 3.14 bugs and security fixes topic-subinterpreters labels Nov 2, 2025

ZeroIntensity reviewed Nov 2, 2025
View reviewed changes

Modules/_struct.c Outdated Show resolvedHide resolved

Use Py_UNUSED for unused arg in init_endian_tables
3130512

ashm-dev force-pushed the tsan branch from 68ffe6d to 3130512Compare November 2, 2025 16:28

ashm-dev requested a review from ZeroIntensity November 2, 2025 16:47

ZeroIntensity reviewed Nov 2, 2025
View reviewed changes

picnixz reviewed Nov 2, 2025
View reviewed changes

Modules/_struct.c Outdated Show resolvedHide resolved

fix: remove duplicate include of pycore_lock.h in _struct.c
d33514b

ashm-dev requested a review from ZeroIntensity November 3, 2025 14:31

ZeroIntensity reviewed Nov 3, 2025
View reviewed changes

Lib/test/test_struct.py Outdated Show resolvedHide resolved

add test
c4f577f

ashm-dev force-pushed the tsan branch from 219b7df to c4f577fCompare November 3, 2025 16:13

ashm-dev requested a review from ZeroIntensity November 3, 2025 16:22

ZeroIntensity reviewed Nov 3, 2025
View reviewed changes

Lib/test/test_struct.py Outdated Show resolvedHide resolved
Lib/test/test_struct.py Outdated Show resolvedHide resolved
Lib/test/test_struct.py Outdated Show resolvedHide resolved
Modules/_struct.c Outdated Show resolvedHide resolved

Remove return check for endian table init
238c40e

ashm-dev force-pushed the tsan branch from e21838a to 238c40eCompare November 3, 2025 16:39

Use InterpreterPoolExecutor in struct tests
f4d24ff

ashm-dev requested review from ZeroIntensity and picnixz November 3, 2025 16:52

ZeroIntensity reviewed Nov 3, 2025
View reviewed changes

Lib/test/test_struct.py Outdated Show resolvedHide resolved
Lib/test/test_struct.pyShow resolvedHide resolved
Lib/test/test_struct.py Outdated Show resolvedHide resolved

ZeroIntensity reviewed Nov 3, 2025
View reviewed changes

Tools/c-analyzer/cpython/globals-to-fix.tsvShow resolvedHide resolved

ZeroIntensity approved these changes Nov 5, 2025
View reviewed changes

bedevere-appbot added awaiting merge and removed awaiting review labels Nov 5, 2025

Merge branch 'main' into tsan
434dbab

kumaraditya303 approved these changes Nov 6, 2025
View reviewed changes

kumaraditya303 enabled auto-merge (squash) November 6, 2025 05:54

ZeroIntensity disabled auto-merge November 11, 2025 20:31

fix: tests
3844826

ashm-dev requested review from ZeroIntensity and kumaraditya303 November 13, 2025 09:29

Merge branch 'main' into tsan
b825107

kumaraditya303 merged commit 63548b3 into python:mainNov 13, 2025
46 checks passed

bedevere-appbot removed the awaiting merge label Nov 13, 2025

bedevere-appbot removed the needs backport to 3.14 bugs and security fixes label Nov 13, 2025

ashm-dev deleted the tsan branch November 13, 2025 11:03

StanFromIreland pushed a commit to StanFromIreland/cpython that referenced this pull request Dec 6, 2025
pythongh-140260: fix data race in_structmodule initialization wit…
1ce571b
…h subinterpreters (python#140909)

Uh oh!

gh-140260: Fix data race in _struct module initialization with subinterpreters#140909

gh-140260: Fix data race in _struct module initialization with subinterpreters #140909

Uh oh!

Conversation

ashm-dev commented Nov 2, 2025• edited by bedevere-app botLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZeroIntensity commented Nov 5, 2025

Uh oh!

ZeroIntensity left a comment

Choose a reason for hiding this comment

Uh oh!

ZeroIntensity commented Nov 5, 2025

Uh oh!

encukou commented Nov 6, 2025

Uh oh!

kumaraditya303 commented Nov 11, 2025

Uh oh!

ZeroIntensity commented Nov 11, 2025

Uh oh!

Uh oh!

miss-islington-appbot commented Nov 13, 2025

Uh oh!

bedevere-appbot commented Nov 13, 2025

Uh oh!

bedevere-bot commented Nov 13, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

mhsmith commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ashm-dev commented Nov 2, 2025•
edited by bedevere-app bot
Loading