gh-138122: Add --subprocesses flag to profile child processes in tachyon#142636
Uh oh!
There was an error while loading. Please reload this page.
Conversation
The _GNU_SOURCE macro must be defined before any system headers are included to enable GNU extensions like process_vm_readv. Moving it before the extern "C" block ensures it takes effect. The internal Python headers are also changed from angle brackets to quotes since they're local to the project. On macOS, the TARGET_OS_OSX macro may not be defined by older SDKs, so we now include TargetConditionals.h explicitly and provide a fallback definition when needed.
This implements platform-specific child process enumeration for use by the profiler. On Linux it parses /proc/{pid}/stat to build a parent- child map and then walks the tree from the target PID. On macOS it uses proc_listchildpids() when available, falling back to scanning all processes with proc_pidinfo(). On Windows it uses CreateToolhelp32Snapshot with TH32CS_SNAPPROCESS to iterate through all processes. The function returns a Python list of PIDs representing all descendants of the given process. The recursive parameter controls whether only direct children or all descendants are returned. This is the building block needed for the --children flag in the sampling profiler CLI.
The ChildProcessMonitor class runs a background thread that polls for new child processes spawned by the target. When it finds a new Python process, it launches a separate profiler subprocess with the same sampling options. Each child profiler writes to its own output file with the child's PID appended to the filename pattern. Detection uses a fast path on Linux (checking /proc/{pid}/exe for "python" in the name) before falling back to the full RemoteUnwinder probe. Non-Python children are silently skipped. There's a limit of 100 concurrent child profilers to avoid runaway resource usage if the target forks heavily. The --children flag is incompatible with --live mode since the curses interface can't handle multiple profiler outputs simultaneously.The test suite covers the get_child_pids C function with both recursive and non-recursive enumeration, the is_python_process detection helper, the ChildProcessMonitor lifecycle, and end-to-end CLI tests with --children for both attach and run modes. Tests use short-lived subprocesses with controlled lifecycles and polling-based synchronization rather than fixed sleeps to keep runtime reasonable. Resource cleanup is handled through reap_children and explicit process termination in finally blocks.
pablogsalforce-pushed the tachyon-subprocesses-atomic branch from 2d7e614 to e6ca9f9Comparepablogsal requested review from StanFromIreland, ivonastojanovic and savannahostrowski
pablogsal commented Dec 12, 2025
CC: @lkollar |
pablogsalforce-pushed the tachyon-subprocesses-atomic branch from ebdb847 to bc5dc46Comparepablogsal changed the title I know this looks like "Oh no another 2k line PR from Pablo 😆 " (and it kind of is) but a lot of this is autogenerated files, tests and docs :) |
pablogsal requested review from a team, AA-Turner, emmatyping and erlend-aasland as code owners
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
When you're done making the requested changes, leave the comment: |
Uh oh!
There was an error while loading. Please reload this page.
pablogsalforce-pushed the tachyon-subprocesses-atomic branch from 9ffd19d to 42d5119Comparepablogsal commented Dec 14, 2025
I have made the requested changes; please review again |
Thanks for making the requested changes! @savannahostrowski: please review the changes made to this pull request. |
savannahostrowski left a comment
savannahostrowski left a comment There was a problem hiding this comment.
One small comment but otherwise LGTM!
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Co-authored-by: Savannah Ostrowski <[email protected]>
pablogsal added the 🔨 test-with-buildbots 
bedevere-bot commented Dec 14, 2025
🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 50b7c89 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F142636%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
bedevere-bot removed the 🔨 test-with-buildbots Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
There is a bug when an output directory is passed:
$ ./python -m profiling.sampling run --subprocesses --flamegraph -o /tmp/dir/ tscript.py [...] IsADirectoryError: [Errno 21] Is a directory: '/tmp/dir/' $ ls dir/ _842702 _842703 _842704 _842705 The output for subprocesses is generated.
Uh oh!
There was an error while loading. Please reload this page.
pablogsal commented Dec 15, 2025
Fixed |
pablogsal added the 🔨 test-with-buildbots bedevere-bot commented Dec 15, 2025
🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 82fe5c8 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F142636%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
bedevere-bot removed the 🔨 test-with-buildbots pablogsal merged commit 6658e2c into python:mainUh oh!
There was an error while loading. Please reload this page.
4 participants
This adds the ability to automatically profile child processes spawned by the target when using the sampling profiler. When the
--subprocessesflag is passed, a background monitor thread polls for new descendants of the target process and spawns separate profiler instances for each Python child it discovers. Each child profiler inherits the sampling options from the parent (interval, duration, thread selection, native frames, async-aware mode, output format) and writes to its own output file with the child's PID appended to the filename.There is a limit of 100 concurrent child profilers to prevent resource exhaustion when profiling applications that fork heavily. The
--subprocessesflag is incompatible with--livemode since the curses interface cannot accommodate multiple concurrent profiler displays. This is useful for profiling applications that usemultiprocessing,ProcessPoolExecutor, or other subprocess-based parallelism.