gh-135953: Add Gecko reporter to sampling profiler

[ivonastojanovic]

Issue: #135953

• Issue: Expose internal stack introspection APIs as a statistical runtime analysis tool #135953

[pablogsal]

CC @lkollar

[Copilot]

Pull Request Overview

This PR adds Gecko reporter functionality to the sampling profiler, enabling export of profiling data in Gecko Profile format for web-based visualization tools. The implementation includes a new GeckoCollector class and supporting data structures.

• Adds new gecko_format.py module with dataclasses and builder for Gecko profile format
• Implements GeckoCollector class in stack_collector.py that exports profiling data as JSON
• Updates sample.py to support --gecko command line option and integrate the new collector

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
Lib/profiling/sampling/gecko_format.py	New module defining Gecko profile format dataclasses and builder
Lib/profiling/sampling/stack_collector.py	Adds GeckoCollector class and category constants
Lib/profiling/sampling/sample.py	Integrates Gecko collector option and updates CLI help text

Comments suppressed due to low confidence (1)

Lib/profiling/sampling/stack_collector.py:1

• GeckoCollector is instantiated without the skip_idle parameter that other collectors receive. Consider whether skip_idle functionality should be supported for consistency with other collectors.

import base64 

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[pablogsal]

There are several problems here:

• The file doesn't really work. If you try to lead it into the firefox profiler it just fails with "Couldn’t read the file or parse the profile in it".
• Generating the profiler with ./python -m profiling.sampling -i 1 -d100000 --realtime-stats --all-threads --gecko -o flamegraph.gecko -m test test_asyncio takes 2GB of memory. That's unfortunately not acceptable
• Generating the profiler when the process ends is super slow (takes 40 seconds on my machine).

[pablogsal]

This also will need some tests

[pablogsal]

@ivonastojanovic I have pushed a draft version of a simpler collector that uses raw dictionaries. This can be processed by the firefox profiler, doesn't blow up the memory in my experiments and also is relatively faster. It needs some polishing but should cover most problems.

[pablogsal]

Great work @ivonastojanovic !

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Ubuntu Shared 3.x (tier-1) has failed when building commit 75b1afe.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/506/builds/11528) and take a look at the build logs.
4. Check if the failure is related to this commit (75b1afe) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/506/builds/11528

Failed tests:

• test_profiling
• test_pyrepl

Failed subtests:

• test_sample_target_script - test.test_profiling.test_sampling_profiler.TestSampleProfilerIntegration.test_sample_target_script
• test_repl_eio - test.test_pyrepl.test_unix_console.TestUnixConsoleEIOHandling.test_repl_eio

Summary of the results of the build (if available):

==

Click to see traceback logs

Traceback (most recent call last): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1137, in __del__ _warn("subprocess %s is still running"%self.pid, ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ResourceWarning, source=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ResourceWarning: subprocess 3224727 is still running Warning -- Unraisable exception Exception ignored while finalizing file <_io.FileIO name=8 mode='rb' closefd=True>: Traceback (most recent call last): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test regrtest_runner(result, test_func, runtests) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner test_result = test_func() File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_funcreturn run_unittest(test_mod, runtests) File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittestreturn _run_suite(tests) File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suiteraise support.TestFailedWithDetails(err, errors, failures, stats=stats) Traceback (most recent call last): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_profiling/test_sampling_profiler.py", line 1924, in test_sample_target_scriptself.assertIn("slow_fibonacci", output) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 'slow_fibonacci' not found in 'Captured 9986 samples in 1.00 seconds\nSample rate: 9985.99 samples/sec\nError rate: 0.00%\nWarning: missed 14 samples from the expected total of 10000 (0.14%)\n\x1b[1;34mProfile Stats:\x1b[0m\n\x1b[1;34m nsamples\x1b[0m \x1b[1;34m sample%\x1b[0m \x1b[1;34mtottime (ms)\x1b[0m \x1b[1;34m cumul%\x1b[0m \x1b[1;34mcumtime (ms)\x1b[0m \x1b[1;34mfilename:lineno(function)\x1b[0m\n 0/9986 0.0 0.000 100.0 998.600 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m63\x1b[0m(\x1b[36m<module>\x1b[0m)\n 0/9918 0.0 0.000 99.3 991.800 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m52\x1b[0m(\x1b[36mmain_loop\x1b[0m)\n 5005/5005 50.1 500.500 50.1 500.500 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m15\x1b[0m(\x1b[36mcpu_intensive_work\x1b[0m)\n 3232/3232 32.4 323.200 32.4 323.200 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m16\x1b[0m(\x1b[36mcpu_intensive_work\x1b[0m)\n 1651/1651 16.5 165.100 16.5 165.100 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m14\x1b[0m(\x1b[36mcpu_intensive_work\x1b[0m)\n 1/68 0.0 0.100 0.7 6.800 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m40\x1b[0m(\x1b[36mnested_calls\x1b[0m)\n 0/68 0.0 0.000 0.7 6.800 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m58\x1b[0m(\x1b[36mmain_loop\x1b[0m)\n 0/67 0.0 0.000 0.7 6.700 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m38\x1b[0m(\x1b[36mnested_calls.<locals>.level1.<locals>.level2\x1b[0m)\n 0/67 0.0 0.000 0.7 6.700 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m39\x1b[0m(\x1b[36mnested_calls.<locals>.level1\x1b[0m)\n 42/42 0.4 4.200 0.4 4.200 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m24\x1b[0m(\x1b[36mmedium_computation\x1b[0m)\n 30/30 0.3 3.000 0.3 3.000 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m17\x1b[0m(\x1b[36mcpu_intensive_work\x1b[0m)\n 25/25 0.3 2.500 0.3 2.500 \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m23\x1b[0m(\x1b[36mmedium_computation\x1b[0m)\n\n\x1b[1;34mLegend:\x1b[0m\n \x1b[33mnsamples\x1b[0m: Direct/Cumulative samples (direct executing / on call stack)\n \x1b[33msample%\x1b[0m: Percentage of total samples this function was directly executing\n \x1b[33mtottime\x1b[0m: Estimated total time spent directly in this function\n \x1b[33mcumul%\x1b[0m: Percentage of total samples when this function was on the call stack\n \x1b[33mcumtime\x1b[0m: Estimated cumulative time (including time in called functions)\n \x1b[33mfilename:lineno(function)\x1b[0m: Function location and name\n\n\x1b[1;34mSummary of Interesting Functions:\x1b[0m\n\n\x1b[1;34mFunctions with Highest Direct/Cumulative Ratio (Hot Spots):\x1b[0m\n 1.000 direct/cumulative ratio, 99.3% direct samples: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mcpu_intensive_work\x1b[0m)\n 1.000 direct/cumulative ratio, 0.7% direct samples: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mmedium_computation\x1b[0m)\n 0.015 direct/cumulative ratio, 0.0% direct samples: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mnested_calls\x1b[0m)\n\n\x1b[1;34mFunctions with Highest Call Frequency (Indirect Calls):\x1b[0m\n 9986 indirect calls, 100.0% total stack presence: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36m<module>\x1b[0m)\n 9986 indirect calls, 100.0% total stack presence: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mmain_loop\x1b[0m)\n 67 indirect calls, 0.7% total stack presence: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mnested_calls\x1b[0m)\n\n\x1b[1;34mFunctions with Highest Call Magnification (Cumulative/Direct):\x1b[0m\n 68.0x call magnification, 67 indirect calls from 1 direct: \x1b[32mtmpy9ith0_b\x1b[0m:\x1b[33m\x1b[0m(\x1b[36mnested_calls\x1b[0m)\n' Traceback (most recent call last): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 837, in gc_collect gc.collect() ~~~~~~~~~~^^ResourceWarning: unclosed file <_io.FileIO name=8 mode='rb' closefd=True> Warning -- Unraisable exception Exception ignored while finalizing file <_io.FileIO name=10 mode='rb' closefd=True>: Traceback (most recent call last): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test regrtest_runner(result, test_func, runtests) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner test_result = test_func() File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_funcreturn run_unittest(test_mod, runtests) File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittestreturn _run_suite(tests) File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suiteraise support.TestFailedWithDetails(err, errors, failures, stats=stats) Traceback (most recent call last): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test regrtest_runner(result, test_func, runtests) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner test_result = test_func() File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_funcreturn run_unittest(test_mod, runtests) File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittestreturn _run_suite(tests) File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suiteraise support.TestFailedWithDetails(err, errors, failures, stats=stats) Traceback (most recent call last): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1137, in __del__ _warn("subprocess %s is still running"%self.pid, ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ResourceWarning, source=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ResourceWarning: subprocess 3247738 is still running Warning -- Unraisable exception Exception ignored while finalizing file <_io.FileIO name=8 mode='rb' closefd=True>: Traceback (most recent call last): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 165, in _load_run_test regrtest_runner(result, test_func, runtests) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 118, in regrtest_runner test_result = test_func() File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 162, in test_funcreturn run_unittest(test_mod, runtests) File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 42, in run_unittestreturn _run_suite(tests) File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/single.py", line 105, in _run_suiteraise support.TestFailedWithDetails(err, errors, failures, stats=stats) Traceback (most recent call last): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 837, in gc_collect gc.collect() ~~~~~~~~~~^^ResourceWarning: unclosed file <_io.FileIO name=10 mode='rb' closefd=True> Traceback (most recent call last): File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pyrepl/test_unix_console.py", line 362, in test_repl_eio _, err = proc.communicate(timeout=5) # sleep for pty to settle~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1219, in communicate stdout, stderr =self._communicate(input, endtime, timeout) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 2126, in _communicateself._check_timeout(endtime, orig_timeout, stdout, stderr) ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/subprocess.py", line 1266, in _check_timeoutraise TimeoutExpired( ...<2 lines>... stderr=b''.join(stderr_seq) if stderr_seq elseNone) subprocess.TimeoutExpired: Command '['/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/python', '-E', '-S', '/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pyrepl/eio_test_script.py']' timed out after 5 seconds

[pablogsal]

This looks unrelated to this PR as this PR only adds test_gecko_collector_basic