gh-137433: Fix deadlock with stop-the-world and daemon threads#137735
Uh oh!
There was an error while loading. Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
There was a deadlock originally seen by Memray when a daemon thread enabled or disabled profiling while the interpreter was shutting down. I think this could also happen with garbage collection, but I haven't seen that in practice. The daemon thread could be hung while trying acquire the global rwmutex that prevents overlapping global and per-interpreter stop-the-world events. Since it already held the main interpreter's stop-the-world lock, it also deadlocked the main thread, which is trying to perform interpreter finalization. Swap the order of lock acquisition to prevent this deadlock. Additionally, refactor `_PyParkingLot_Park` so that the global buckets hashtable is left in a clean state if the thread is hung in `PyEval_AcquireThread`.
gpshead approved these changes Sep 14, 2025
colesbury requested review from ZeroIntensity and ericsnowcurrently as code owners
ZeroIntensity approved these changes Sep 15, 2025
Member
ZeroIntensity left a comment
ZeroIntensity left a comment There was a problem hiding this comment.
LGTM. I'm not a huge fan of using PyEval_AcquireThread/PyEval_ReleaseThread over _PyThreadState_Attach/_PyThreadState_Detach, but since those were there already I'm not going to bother complaining about it.
colesbury merged commit 90fe325 into python:main 85 of 87 checks passed
Uh oh!
There was an error while loading. Please reload this page.
Thanks @colesbury for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14. |
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Sep 16, 2025
…ythongh-137735) There was a deadlock originally seen by Memray when a daemon thread enabled or disabled profiling while the interpreter was shutting down. I think this could also happen with garbage collection, but I haven't seen that in practice. The daemon thread could be hung while trying acquire the global rwmutex that prevents overlapping global and per-interpreter stop-the-world events. Since it already held the main interpreter's stop-the-world lock, it also deadlocked the main thread, which is trying to perform interpreter finalization. Swap the order of lock acquisition to prevent this deadlock. Additionally, refactor `_PyParkingLot_Park` so that the global buckets hashtable is left in a clean state if the thread is hung in `PyEval_AcquireThread`. (cherry picked from commit 90fe325) Co-authored-by: Sam Gross <[email protected]>
GH-138965 is a backport of this pull request to the 3.14 branch. |
encukou pushed a commit that referenced this pull request Oct 7, 2025
…gh-137735) (GH-138965) There was a deadlock originally seen by Memray when a daemon thread enabled or disabled profiling while the interpreter was shutting down. I think this could also happen with garbage collection, but I haven't seen that in practice. The daemon thread could be hung while trying acquire the global rwmutex that prevents overlapping global and per-interpreter stop-the-world events. Since it already held the main interpreter's stop-the-world lock, it also deadlocked the main thread, which is trying to perform interpreter finalization. Swap the order of lock acquisition to prevent this deadlock. Additionally, refactor `_PyParkingLot_Park` so that the global buckets hashtable is left in a clean state if the thread is hung in `PyEval_AcquireThread`. (cherry picked from commit 90fe325) Co-authored-by: Sam Gross <[email protected]>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There was a deadlock originally seen by Memray when a daemon thread enabled or disabled profiling while the interpreter was shutting down. I think this could also happen with garbage collection, but I haven't seen that in practice.
The daemon thread could be hung while trying acquire the global rwmutex that prevents overlapping global and per-interpreter stop-the-world events. Since it already held the main interpreter's stop-the-world lock, it also deadlocked the main thread, which is trying to perform interpreter finalization.
Swap the order of lock acquisition to prevent this deadlock. Additionally, refactor
_PyParkingLot_Parkso that the global buckets hashtable is left in a clean state if the thread is hung inPyEval_AcquireThread.