Skip to content

Conversation

@lkollar
Copy link
Contributor

@lkollarlkollar commented Sep 13, 2025

The stack collector base class keeps all frames until export() is called, which causes significant unnecessary memory usage. Instead, we can process the frames on the fly in the collect call by dispatching the aggregation logic to the subclass through the process_frames method.

This comment was marked as off-topic.

@pythonpython deleted a comment from CopilotAISep 13, 2025
@pythonpython deleted a comment from CopilotAISep 13, 2025
@pablogsalpablogsalforce-pushed the profiling/reduce-memory-usage branch 2 times, most recently from eb1bffb to a89c852CompareSeptember 14, 2025 02:51
@pablogsal
Copy link
Member

pablogsal commented Sep 14, 2025

Amazing! I have pushed a commit adding an initial version of the string table for the flame graph. We can improve that in subsequent PRs

@pablogsalpablogsal enabled auto-merge (squash) September 14, 2025 20:52
@pablogsal
Copy link
Member

Will fix merge conflicts tomorrow

lkollarand others added 2 commits September 14, 2025 23:17
The stack collector base class keeps all frames until export() is called, which causes significant unnecessary memory usage. Instead, we can process the frames on the fly in the collect call by dispatching the aggregation logic to the subclass through the process_frames method.
@pablogsalpablogsalforce-pushed the profiling/reduce-memory-usage branch from a89c852 to 52c9867CompareSeptember 14, 2025 22:18
@pablogsalpablogsal merged commit 3e06cfc into python:mainSep 14, 2025
43 checks passed
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 CentOS9 NoGIL Refleaks 3.x (tier-1) has failed when building commit 3e06cfc.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1610/builds/2071) and take a look at the build logs.
  4. Check if the failure is related to this commit (3e06cfc) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1610/builds/2071

Failed tests:

  • test_free_threading

Test leaking resources:

  • test_free_threading: file descriptors

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 22, done. remote: Counting objects: 4% (1/22) remote: Counting objects: 9% (2/22) remote: Counting objects: 13% (3/22) remote: Counting objects: 18% (4/22) remote: Counting objects: 22% (5/22) remote: Counting objects: 27% (6/22) remote: Counting objects: 31% (7/22) remote: Counting objects: 36% (8/22) remote: Counting objects: 40% (9/22) remote: Counting objects: 45% (10/22) remote: Counting objects: 50% (11/22) remote: Counting objects: 54% (12/22) remote: Counting objects: 59% (13/22) remote: Counting objects: 63% (14/22) remote: Counting objects: 68% (15/22) remote: Counting objects: 72% (16/22) remote: Counting objects: 77% (17/22) remote: Counting objects: 81% (18/22) remote: Counting objects: 86% (19/22) remote: Counting objects: 90% (20/22) remote: Counting objects: 95% (21/22) remote: Counting objects: 100% (22/22) remote: Counting objects: 100% (22/22), done. remote: Compressing objects: 12% (1/8) remote: Compressing objects: 25% (2/8) remote: Compressing objects: 37% (3/8) remote: Compressing objects: 50% (4/8) remote: Compressing objects: 62% (5/8) remote: Compressing objects: 75% (6/8) remote: Compressing objects: 87% (7/8) remote: Compressing objects: 100% (8/8) remote: Compressing objects: 100% (8/8), done. remote: Total 12 (delta 9), reused 5 (delta 4), pack-reused 0 (from 0)  From https://github.com/python/cpython * branch main -> FETCH_HEAD Note: switching to '3e06cfcaeee31c2a6e9befe5ea54d9beaa2c0434'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c <new-branch-name> Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at 3e06cfcaeee gh-135953: Reduce memory usage of stack collectors (#138875) Switched to and reset branch 'main' configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal modulemake: *** [Makefile:2486: buildbottest] Error 2

@lkollarlkollar deleted the profiling/reduce-memory-usage branch September 22, 2025 17:51
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@lkollar@pablogsal@bedevere-bot