REPL: AttributeError: module `__mp_main__` has no attribute `is_prime` in `ProcessPoolExecutor` example

[wenming21]

Bug report

Bug description:

python version:3.12.9
An error occurred when running as a standalone script.
An error message is reported when running the sample code of the python document:

Traceback (most recent call last):
File "C:\Users\wen\Desktop\test.py", line 32, in
main()
File "C:\Users\wen\Desktop\test.py", line 28, in main
for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Miniconda3\Lib\concurrent\futures\process.py", line 636, in _chain_from_iterable_of_lists
for element in iterable:
^^^^^^^^
File "C:\Miniconda3\Lib\concurrent\futures_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Miniconda3\Lib\concurrent\futures_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "C:\Miniconda3\Lib\concurrent\futures_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Miniconda3\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

import concurrent.futures
import math

PRIMES = [
 112272535095293,
 112582705942171,
 112272535095293,
 115280095190773,
 115797848077099,
 1099726899285419]

def is_prime(n):
 if n < 2:
 return False
 if n == 2:
 return True
 if n % 2 == 0:
 return False

 sqrt_n = int(math.floor(math.sqrt(n)))
 for i in range(3, sqrt_n + 1, 2):
 if n % i == 0:
 return False
 return True

def main():
 with concurrent.futures.ProcessPoolExecutor() as executor:
 for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
 print('%d is prime: %s' % (number, prime))

if __name__ == '__main__':
 main()

CPython versions tested on:

3.12

Operating systems tested on:

Windows 10

Linked PRs

• gh-132898: Add a note in multiprocessing.Process docs about creating a process in a REPL session #137118
• [3.14] gh-132898: Add a note in multiprocessing.Process docs about creating a process in a REPL (GH-137118) #137154
• [3.13] gh-132898: Add a note in multiprocessing.Process docs about creating a process in a REPL (GH-137118) #137155

[picnixz]

Question: did this occur because you hit Ctrl+C or killed the process somehow?

Nevermind, it occurred on Linux as well with just running the example.

[picnixz]

Ok, I only found the issue when the above code is being tested inside the REPL and not as a script (namely python test.py should work but not when I C/C the above code). The best fix for now would be to just indicate in the docs "run it as a standalone script, not inside the REPL", but I don't know the exact reason why (well I can guess: it has something with pickling and probably globals that are not made available because they are only available in the REPL and ProcessPoolExecutor is not able to find the correct "main" module).

cc @pablogsal @gpshead

[picnixz]

@wenming21 Your latest edit indicates that the error also occurs when running the example as a standalone script. What is the full traceback? (namely, what's above the "Traceback (most recent call last):"? because I do have other stuff written when I encounter the bug)

[StanFromIreland]

Why was this closed? @wenming21

[StanFromIreland]

I can also add that this only occurs on the main branch, on 3.13 it works fine.

[picnixz]

3.14 has seen changes with the default start method so this could be the cause by the way (at least on Linux)

[YvesDup]

What's happen when you change the start_method ?
I am on Mac, with a 3.12.8 version and I ran that script:
/bin/python3 -i /Users/yves/Desktop/Bugs/gh-132898.py and in REPL, I call main(), I have got errors:

Ouptput

multiprocessing.get_start_method() = 'spawn' # I print the current `start_method` and keep only two numbers to check
112272535095293 is prime: True
1099726899285419 is prime: False
>>> main()
Process SpawnProcess-4:
Traceback (most recent call last):
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
 self.run()
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/process.py", line 108, in run
 self._target(*self._args, **self._kwargs)
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/process.py", line 252, in _process_worker
 call_item = call_queue.get(block=True)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/queues.py", line 122, in get
 return _ForkingPickler.loads(res)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'is_prime' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
Process SpawnProcess-3:
Traceback (most recent call last):
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
 self.run()
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/process.py", line 108, in run
 self._target(*self._args, **self._kwargs)
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/process.py", line 252, in _process_worker
 call_item = call_queue.get(block=True)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/queues.py", line 122, in get
 return _ForkingPickler.loads(res)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'is_prime' on <module '__main__' (<class '_frozen_importlib.BuiltinImporter'>)>
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/Users/yves/Desktop/Bugs/gh-132898.py", line 31, in main
 for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/process.py", line 636, in _chain_from_iterable_of_lists
 for element in iterable:
 ^^^^^^^^
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 619, in result_iterator
 yield _result_or_cancel(fs.pop())
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 317, in _result_or_cancel
 return fut.result(timeout)
 ^^^^^^^^^^^^^^^^^^^
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 456, in result
 return self.__get_result()
 ^^^^^^^^^^^^^^^^^^^
 File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
 raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while

if I changed the start_method to forkserver, I have got the same errors.

And finally if the start method is fork everything is fine.
/usr/local/bin/python3 -i /Users/yves/Desktop/Bugs/gh-132898.py

multiprocessing.get_start_method() = 'fork'
112272535095293 is prime: True
1099726899285419 is prime: False
>>> main()
112272535095293 is prime: True
1099726899285419 is prime: False
>>> 

IMO, this issue is also causing by start_method but I don't figure out how REPL is involved.
And a last remark, if you replace the is_prime function with math.sqrt, the script succeeds in any start_method case .