multi-threading + fork warning when threads are stopped before fork

[P403n1x87]

Bug report

Bug description:

If all threads are stopped before a fork, and then restarted in the parent (and optionally the child) process, the new multi-threading with fork warning is given

import os
import threading


stop = False


def target():
 while not stop:
 pass


thread = threading.Thread(target=target)


def stop_thread():
 global stop
 stop = True
 thread.join()


def restart_thread():
 global thread, stop
 stop = False
 (thread := threading.Thread(target=target)).start()


os.register_at_fork(before=stop_thread, after_in_child=restart_thread, after_in_parent=restart_thread)


thread.start()

if os.fork() == 0:
 os._exit(0)

If the thread is not restarted in the parent, no warning is given. I believe warn_about_fork_with_threads should be refactored so that a warning is given to the user only if there are threads detected after the call to PyOS_BeforeFork. If it is not possible to give a warning there, as stated in the comments, perhaps one can just do the thread count check there and take note of the result, then warn after PyOS_AfterFork_Parent, when safe to do so.

CPython versions tested on:

3.13

Operating systems tested on:

macOS

Linked PRs

• gh-137109: refactor warning about threads when forking #141438
• [3.14] gh-137109: refactor warning about threads when forking (GH-141438) #141614
• [3.13] gh-137109: refactor warning about threads when forking (GH-141438) (GH-141614) #141639

[picnixz]

Let's remove the callback in the child. The order of execution would be:

• Start thread.
• Request to fork.
• Join thread.
• Do the fork.
• Restart the thread in the parent.

I think the problem is with the fact that thread from the parent process is not entirely joined. And so it's effectively the case that we're still multi-threaded even though it's not the case maybe.

cc @ZeroIntensity as someone who has better threading knowledge.

[P403n1x87]

It looks to me that warn_about_fork_with_threads is warning on the new thread in the parent, because it is restarted by PyOS_AfterFork_Parent, before warn_about_fork_with_threads is called. So the warning is about the new thread running in the parent after the fork, rather than the original thread (that has actually been stopped and joined before the call to the fork syscall).

[colesbury]

Yeah, it'd be better to call the after-fork-parent calls after we check and warn about threads. That would avoid the spurious warning.

[gpshead]

Code paths that warn_about_fork_with_thread can take depend on PyOS_AfterFork_Parent() having been called already to avoid deadlock.

[gpshead]

A workaround for this could be for it to be called by PyOS_AfterFork_Parent() itself right before the run_at_forkers(...) call.

[gpshead]

But that'd still leave deadlocks possible as the Python world it may need to call into could rely on other afterfork parent calls that have been registered.

[gpshead]

Okay I think I have a solution for the most common case. The warn function can be split up into the OS specific APIs to get the number of threads (which can be called before AfterFork_Parent) and the Python APIs that must wait until after. This won't fix it in situations where to lack platform specific thread detection code or when those APIs fail, but will in the most widely used scenarios. PR coming after testing.

[gpshead]

re: race conditions, it may be OS specific as to if the thread could still show up in OS APIs as existing in the process after the internal C pthread_join() that the Python thread.join() did has returned. I'd like to think not, but that is up to the OS as there are no posix APIs to get a list of threads for a process so it is undefined behavior and up to the kernel implementation as to if it has finished tearing down that state and cleaning it up synchronously or not.