Improve documentation on multiprocessing synchronization-between-processes

[HCharlie]

Documentation

https://docs.python.org/3/library/multiprocessing.html#synchronization-between-processes
(A clear and concise description of the issue.)

when running the provided example locally, unlike other examples on this page which can print valid stdout, the code in this section will only print out the error below, which is caused by not providing the Process.join() call at the end.

Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "/Users/chhan/.pyenv/versions/3.12.2/lib/python3.12/multiprocessing/spawn.py", line 122, in spawn_main
 exitcode = _main(fd, parent_sentinel)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Users/chhan/.pyenv/versions/3.12.2/lib/python3.12/multiprocessing/spawn.py", line 132, in _main
 self = reduction.pickle.load(from_parent)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Users/chhan/.pyenv/versions/3.12.2/lib/python3.12/multiprocessing/synchronize.py", line 115, in __setstate__
 self._semlock = _multiprocessing.SemLock._rebuild(*state)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory

An example patch could be

from multiprocessing import Process, Lock

 def f(l, i):
 l.acquire()
 try:
 print('hello world', i)
 finally:
 l.release()

 if __name__ == '__main__':
 lock = Lock()

 processes = []

 for num in range(10):
 p = Process(target=f, args=(lock, num))
 processes.append(p)
 p.start()
 
 for p in processes:
 p.join()

Linked PRs

• GH-116526 patch documentation Synchronization between processes example #116527

[gaogaotiantian]

To be honest this is a pretty bad example to begin with - because you won't notice any difference if you simply remove the lock so it's pretty confusing when users try it out. The stdout is buffered in most environments and won't flush until seeing a newline. We should probably change this to something more obvious - multiple prints maybe (building a triangle?).

The exception mentioned in this issue can be reproduced with spawn, but might be hard to repro with fork as fork is too fast.

[HCharlie]

Hey @gaogaotiantian , thanks for your reply. I think your point makes sense, and it still confuses me when reading it as well. Either with or without the lock, the example doesn't seem to emphasize what difference this lock brings. Tbh, my original intent was to just complete the example to remove some exceptions, now I think maybe a new example is needed.

[HCharlie]

one quick example I could think of is as below, we increment a value 3 times in a process.

import multiprocessing

# Function to be executed by each process
def increment_counter(counter, lock, process_id):
 lock.acquire()
 for _ in range(3):
 counter.value += 1
 print(f"Counter value: {counter.value} incremented by the {process_id} process")
 lock.release()

if __name__ == "__main__":
 # Shared counter value
 counter = multiprocessing.Value('i', 0)
 
 # Creating a Lock
 lock = multiprocessing.Lock()

 # Creating processes
 processes = []
 for j in range(4): # Creating 4 processes
 p = multiprocessing.Process(target=increment_counter, args=(counter, lock, j))
 processes.append(p)
 p.start()

 # Waiting for all processes to finish
 for p in processes:
 p.join()

 # Output the final value of the counter
 print("Final counter value:", counter.value)

with the lock, the value will consecutively be incremented by the same process, and the example will output something like this

Counter value: 1 incremented by the 1 process
Counter value: 2 incremented by the 1 process
Counter value: 3 incremented by the 1 process
Counter value: 4 incremented by the 0 process
Counter value: 5 incremented by the 0 process
Counter value: 6 incremented by the 0 process
Counter value: 7 incremented by the 2 process
Counter value: 8 incremented by the 2 process
Counter value: 9 incremented by the 2 process
Counter value: 10 incremented by the 3 process
Counter value: 11 incremented by the 3 process
Counter value: 12 incremented by the 3 process
Final counter value: 12

if we remove the lock-related code,

import multiprocessing

# Function to be executed by each process
def increment_counter(counter, process_id):

 for _ in range(3):
 counter.value += 1
 print(f"Counter value: {counter.value} incremented by the {process_id} process")


if __name__ == "__main__":
 # Shared counter value
 counter = multiprocessing.Value('i', 0)

 # Creating processes
 processes = []
 for j in range(4): # Creating 4 processes
 p = multiprocessing.Process(target=increment_counter, args=(counter, j))
 processes.append(p)
 p.start()

 # Waiting for all processes to finish
 for p in processes:
 p.join()

 # Output the final value of the counter
 print("Final counter value:", counter.value)

it will randomly print out some duplicates like below, eventually not even reaching 12.

Counter value: 1 incremented by the 3 process
Counter value: 1 incremented by the 2 process
Counter value: 1 incremented by the 0 process
Counter value: 2 incremented by the 1 process
Counter value: 3 incremented by the 3 process
Counter value: 4 incremented by the 2 process
Counter value: 4 incremented by the 0 process
Counter value: 4 incremented by the 1 process
Counter value: 5 incremented by the 3 process
Counter value: 6 incremented by the 2 process
Counter value: 6 incremented by the 0 process
Counter value: 6 incremented by the 1 process
Final counter value: 6

wdyt?