3.15 profiling, can't use ProcessPoolExecutor.map() in a simple script

[dimaqq]

Bug report

Bug description:

# test/test_unit.py
import concurrent.futures
...

def exercise(units=11, rounds=2):
 with concurrent.futures.ProcessPoolExecutor() as executor:
 ...
 results = list(
 executor.map(
 step,
 [f"app/{i}" for i in range(units)],
 [config] * units,
 [local_app_data] * units,
 [peers_data] * units,
 )
 )


def step(
 unit: str,
 config: Mapping[str, str | int | float | bool],
 local_app_data: Mapping[str, JSON],
 all_units_data: Mapping[ops.testing.UnitID, Mapping[str, JSON]],
) -> tuple[dict[str, JSON] | None, dict[str, JSON], str | None, str]:
 ...

if __name__ == "__main__":
 import sys

 exercise(int(sys.argv[1]), int(sys.argv[2]))

(jgol-peer) 🦐/c/j/peer (main)> sudo -E env PYTHONPATH=src python -m profiling.sampling test/test_unit.py 100 100
Profiler coordinator error: Error executing script '/code/jgol/peer/test/test_unit.py': Can't pickle <function step at 0x102b7aae0>: it's not found as __main__.step

CPython versions tested on:

3.15

Operating systems tested on:

macOS

Linked PRs

• gh-140729: Add __mp_main__ as a duplicate for __main__ for pickle to work #140735
• gh-140729: Fix subprocess handling in test_process_pool_executor_pickle #141688
• gh-140729: profiling or tracing multiprocessing can cause error #141874

[picnixz]

cc @pablogsal

[yihong0618]

This is interesting
its very similar like viztracer issue which is gaogaotiantian/viztracer#423
in ProcessPoolExecutor() the __main__ may because __mp_main__

a simple reproducer

cat ttt.py

import concurrent.futures


def worker(x):
 return x * 2


if __name__ == "__main__":
 with concurrent.futures.ProcessPoolExecutor() as executor:
 results = list(executor.map(worker, [1, 2, 3]))
 print(f"Results: {results}")

➜ cpython git:(main) ✗ sudo ./python.exe -m profiling.sampling ttt.py 100 100
Profiler coordinator error: Error executing script '/Users/hyi/prs/cpython/ttt.py': Can't pickle <function worker at 0x109a2d170>: it's not found as __main__.worker
Captured 1000 samples in 0.10 seconds
Sample rate: 10000.00 samples/sec
Error rate: 0.60%
Profile Stats:
 nsamples sample% tottime (ms) cumul% cumtime (ms) filename:lineno(function)
 0/606 0.0 0.000 67.3 60.600 _base.py:694(Executor.__exit__)
 604/604 67.0 60.400 67.0 60.400 threading.py:1126(Thread.join)
 0/604 0.0 0.000 67.0 60.400 process.py:865(ProcessPoolExecutor.shutdown)
 0/143 0.0 0.000 15.9 14.300 <frozen importlib._bootstrap>:1379(_find_and_load)
 0/137 0.0 0.000 15.2 13.700 <frozen importlib._bootstrap>:1350(_find_and_load_unlocked)
 0/78 0.0 0.000 8.7 7.800 <frozen importlib._bootstrap>:938(_load_unlocked)
 0/77 0.0 0.000 8.5 7.700 <frozen importlib._bootstrap_external>:759(_LoaderBasics.exec_module)
 0/59 0.0 0.000 6.5 5.900 <frozen importlib._bootstrap>:924(_load_unlocked)
 1/58 0.1 0.100 6.4 5.800 <frozen importlib._bootstrap_external>:855(SourceLoader.get_code)
 58/58 6.4 5.800 6.4 5.800 <frozen importlib._bootstrap>:491(_call_with_frames_removed)
 0/57 0.0 0.000 6.3 5.700 <frozen importlib._bootstrap_external>:1057(ExtensionFileLoader.create_module)
 0/57 0.0 0.000 6.3 5.700 <frozen importlib._bootstrap>:816(module_from_spec)
 50/50 5.5 5.000 5.5 5.000 <frozen importlib._bootstrap_external>:955(FileLoader.get_data)
 31/31 3.4 3.100 3.4 3.100 resource_tracker.py:114(ResourceTracker._stop_locked)
 0/31 0.0 0.000 3.4 3.100 resource_tracker.py:89(ResourceTracker._stop)

Legend:
 nsamples: Direct/Cumulative samples (direct executing / on call stack)
 sample%: Percentage of total samples this function was directly executing
 tottime: Estimated total time spent directly in this function
 cumul%: Percentage of total samples when this function was on the call stack
 cumtime: Estimated cumulative time (including time in called functions)
 filename:lineno(function): Function location and name

Summary of Interesting Functions:

Functions with Highest Direct/Cumulative Ratio (Hot Spots):
 1.000 direct/cumulative ratio, 67.0% direct samples: threading.py:(Thread.join)
 1.000 direct/cumulative ratio, 6.4% direct samples: <frozen importlib._bootstrap>:(_call_with_frames_removed)
 1.000 direct/cumulative ratio, 5.5% direct samples: <frozen importlib._bootstrap_external>:(FileLoader.get_data)

Functions with Highest Call Frequency (Indirect Calls):
 606 indirect calls, 67.3% total stack presence: _base.py:(Executor.__exit__)
 604 indirect calls, 67.0% total stack presence: process.py:(ProcessPoolExecutor.shutdown)
 143 indirect calls, 15.9% total stack presence: <frozen importlib._bootstrap>:(_find_and_load)

Functions with Highest Call Magnification (Cumulative/Direct):
 58.0x call magnification, 57 indirect calls from 1 direct: <frozen importlib._bootstrap_external>:(SourceLoader.get_code)

the patch can be like what viztracer did

diff --git a/Lib/profiling/sampling/_sync_coordinator.py b/Lib/profiling/sampling/_sync_coordinator.py
index 79e8858ca1..43a80b6adb 100644
--- a/Lib/profiling/sampling/_sync_coordinator.py
+++ b/Lib/profiling/sampling/_sync_coordinator.py
@@ -177,8 +177,14 @@ def _execute_script(script_path: str, script_args: List[str], cwd: str) -> None:
 source_code = f.read()
 
 # Compile and execute the script
+ import types
+ main_module = types.ModuleType("__main__")
+ main_module.__file__ = script_path
+ main_module.__builtins__ = __builtins__
+ sys.modules['__main__'] = sys.modules['__mp_main__'] = main_module
+
 code = compile(source_code, script_path, 'exec')
- exec(code,{'__name__': '__main__', '__file__': script_path})
+ exec(code, main_module.__dict__)
 except FileNotFoundError as e:
 raise TargetError(f"Script file not found:{script_path}") from e
 except PermissionError as e:

after the patch
cc @dimaqq

[mehetabalom181-lab]

Bug report

Bug description:

test/test_unit.py

import concurrent.futures
...

def exercise(units=11, rounds=2):
with concurrent.futures.ProcessPoolExecutor() as executor:
...
results = list(
executor.map(
step,
[f"app/{i}" for i in range(units)],
[config] * units,
[local_app_data] * units,
[peers_data] * units,
)
)

def step(
unit: @str,
config: Mapping[str, str | int | float | bool],
local_app_data: Mapping[str, JSON],
all_units_data: Mapping[ops.testing.UnitID, Mapping[str, JSON]],
) -> tuple[dict[str, JSON] | None, dict[str, JSON], str | None, str]:
...

if name == "main":
import sys

exercise(int(sys.argv[1]), int(sys.argv[2]))

(jgol-peer) 🦐/c/j/peer (main)> sudo -E env PYTHONPATH=src python -m profiling.sampling test/test_unit.py 100 100
Profiler coordinator error: Error executing script '/code/jgol/peer/test/test_unit.py': Can't pickle <function step at 0x102b7aae0>: it's not found as main.step

CPython versions tested on:

3.15

Operating systems tested on:

macOS

Linked PRs

• gh-140729: Add mp_main as a duplicate for main for pickle to work #140735

[YvesDup]

The execution of ttt.py script fails when run via the cProfile module as below:

python -m cProfile /Users/yves/Desktop/Bugs/gh-140729.py
Process SpawnProcess-1:
Traceback (most recent call last):
 File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/process.py", line 320, in _bootstrap
 self.run()
 ~~~~~~~~^^
 File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/process.py", line 108, in run
 self._target(*self._args, **self._kwargs)
 ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/concurrent/futures/process.py", line 242, in _process_worker
 call_item = call_queue.get(block=True)
 File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/queues.py", line 115, in get
 return _ForkingPickler.loads(res)
 ~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: module '__main__' has no attribute 'worker'

and with the profile module too.

Based on my initial tests, the proposed fix is not valid for these two modules. Should we open a new issue ?

[yihong0618]

The execution of ttt.py script fails when run via the cProfile module as below:

python -m cProfile /Users/yves/Desktop/Bugs/gh-140729.py
Process SpawnProcess-1:
Traceback (most recent call last):
 File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/process.py", line 320, in _bootstrap
 self.run()
 ~~~~~~~~^^
 File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/process.py", line 108, in run
 self._target(*self._args, **self._kwargs)
 ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/concurrent/futures/process.py", line 242, in _process_worker
 call_item = call_queue.get(block=True)
 File "/Users/yves/Documents/_Jobs/_Clients/cpython2/cpython/Lib/multiprocessing/queues.py", line 115, in get
 return _ForkingPickler.loads(res)
 ~~~~~~~~~~~~~~~~~~~~~^^^^^
AttributeError: module '__main__' has no attribute 'worker'

and with the profile module too.

Based on my initial tests, the proposed fix is not valid for these two modules. Should we open a new issue ?

using the same method can fix this?

[YvesDup]

using the same method can fix this?

I don`t know yet

[YvesDup]

@picnixz, @StanFromIreland I am wondering if a topic-profiling (if exists) or a topic-multiprocessing label should not be better than topic-subinterpreters.