multiprocessing.Pool map hangs

[wdxal]

Bug report

Bug description:

Q:
Run the following code, sometimes subprogres that is created by [class A's Pool or class B's Pool] is going to sleeping status, and there is not any error log output.
But code（#3 or #4）run success, code(#1 or #2) is not run,The reason is pool.close() and pool.join() is not writed in source code?

test.py

from multiprocessing import Pool
import os
from test_common import *

class Test:
 def __init__(self):
 self.A = A()
 self.B = B()

 def run(self):
 data_list = list(range(10001, 11984))

 a_list = self.A.run(data_list)
 print("self.A.run end") #1

 new_list = self.B.run(data_list, a_list)
 print("self.B.run end") #2


def main():
 run_module = Test()
 run_module.run()

if __name__ == '__main__':
 main()

---------------------------------------------
test_common.py

import os
import multiprocessing
from multiprocessing import Pool
import io
import datetime
import time

class A:
 def __init__(self):
 pass

 def run(self, data_list):
 a_list = []
 for item in data_list:
 file_path = "{}/{}.aaa".format("/tmp/aaa", item)
 a_list.append(file_path)

 pool_size = 1
 pool = Pool(pool_size)
 pool.map(methodA, a_list)
 print("A#run end") #3
 return a_list


def methodA(file_path):
 if os.path.isfile(file_path):
 return
 # other code

class B:
 def __init__(self):
 pass

 def run(self, data_list, a_list): 
 b_list = []
 for item in data_list:
 b_path = "{}/{}.bbb".format("/tmp/bbb", item)
 b_list.append(b_path)

 pool_size = 1
 pool = Pool(pool_size)
 pool.map(methodB, b_list)
 
 print("B#run end") #4
 return b_list

def methodB(b_path):
 if os.path.isfile(b_path):
 return
 # other code

CPython versions tested on:

3.10

Operating systems tested on:

Linux

[ZeroIntensity]

I haven't had a chance to try and reproduce this yet, but does this occur on 3.12+? Things on <3.12 can only get security fixes.

[wdxal]

Thanks for your reply.

Is the lack of close() and join() in my code the reason why this is happening?
In my src code, the probability of occurrence is extremely low.

I haven't had a chance to try and reproduce this yet, but does this occur on 3.12+? Things on <3.12 can only get security fixes.

Sorry, about3.12+，I don't have the opportunity to test it.
Can you tell me which security update it is?

[ZeroIntensity]

Any version below 3.12 cannot get bugfixes like this anymore, they can only be changed for a security problem (which doesn't include this issue, unfortunately). FWIW, I wasn't able to reproduce the issue on 3.12

[picnixz]

Is the lack of close() and join() in my code the reason why this is happening?

Yes, probably. The docs say in https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool:

multiprocessing.pool objects have internal resources that need to be properly managed (like any other resource) by using the pool as a context manager or by calling close() and terminate() manually. Failure to do this can lead to the process hanging on finalization.

The best practice is to use a context manager as in the first docs example:

from multiprocessing import Pool

def f(x):
 return x*x

if __name__ == '__main__':
 with Pool(5) as p:
 print(p.map(f, [1, 2, 3]))

I also cannot reproduce the hanging, even on 3.10 but since the usage of Pool is incorrect, I am closing this one as not an issue.

[wdxal]

@ZeroIntensity @picnixz
Thanks！

In fact, in my environment(3.10), the probability of this problem reproduce is also very low.
After adding the resource closing code, I haven't reproduced this problem so far. It seems to be solved.