GH-127705: Use `_PyStackRef`s in the default build.#127875

markshannon · 2024-12-12T16:28:01Z

We already have _PyStackRefs in the default build, but they are just PyObject * pointers cast to a pointer-sized int.

This PR adds tags according to the tagging scheme in #127705.

Issue: Use tagged pointers on the stack in the default build. #127705

…xes to immortal objects.

…in LOAD_CONST

…e-127705.Qad2hx.rst Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

ericsnowcurrently

This mostly seems okay. FYI, I didn't do a deep dive into the code to verifying correctness, like I normally do. I did read through all the files except for Include/internal/pycore_stackref.h. The changes make sense and I can see how they simplify things.

Include/internal/pycore_frame.h

Include/internal/pycore_stackref.h

mpage · 2025-03-06T02:19:36Z

@markshannon - Have you measured the effect of the mortality optimizations independently from the changes to use macros? The tag bits are a precious commodity and the mortality optimizations require two bits.

markshannon · 2025-03-07T14:53:57Z

@mpage. I changed the PR to use a single bit, using _Py_IsImmortal() in a few places where we need to distinguish between tagged pointers to mortal and immortal objects.
It has surprisingly little effect, the number differ per platform but the net effect is still about 0.

Linux (x86-64) 1.0% slower
MacOS 0.1% faster
Windows 1.1% faster

Full results

Fidget-Spinner · 2025-03-09T18:30:10Z

Python/bytecodes.c

 inst(RETURN_VALUE, (retval--res)){
 assert(frame->owner!=FRAME_OWNED_BY_INTERPRETER);
 _PyStackReftemp=retval;
+assert(PyStackRef_IsHeapSafe(temp));


Why the assert here? This blocks stackrefs from passing through to the host frame, which blocks a lot of unboxed and deferred refcounting optimizations.

It blocks a lot of unsafe operations. Passing references towards the caller is unsafe as it is possible that the only immediate reference count (stored in ob_refcnt) for an object is stored in the callee, and will be deleted during the return.

bedevere-bot · 2025-03-10T15:47:35Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL9 Refleaks 3.x (tier-3) has failed when building commit 2bef8ea.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1589/builds/1084) and take a look at the build logs.
Check if the failure is related to this commit (2bef8ea) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1589/builds/1084

Failed tests:

test_import

Test leaking resources:

test_xml_etree_c: references
test_ast: references
test___all__: references
test_bdb: references
test_pydoc: references
test_import: memory blocks
test_pyclbr: references
test_zipfile: references
test_idle: references
test_datetime: references
test_import: references

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_exec_failure) ... ok TracebackTests.test_syntax_error) ... ok TracebackTests.test_broken_from) ... ok TracebackTests.test_broken_submodule) ... ok TracebackTests.test_broken_parent) ... ok TracebackTests.test_unencodable_filename) ... ok TracebackTests.test_exec_failure_nested) ... ok TracebackTests.test_import_bug) ... ok TracebackTests.test_broken_parent_from) ... ok TracebackTests.test_nonexistent_module) ... ok TracebackTests.test_nonexistent_module_nested) ... ok

bedevere-bot · 2025-03-10T15:49:21Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 Refleaks 3.x (tier-3) has failed when building commit 2bef8ea.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/75/builds/2284) and take a look at the build logs.
Check if the failure is related to this commit (2bef8ea) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/75/builds/2284

Failed tests:

test_import

Test leaking resources:

test___all__: references
test_timeit: references
test_import: memory blocks
test_zipfile: references
test_idle: references
test_import: references

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_exec_failure) ... ok TracebackTests.test_syntax_error) ... ok TracebackTests.test_broken_from) ... ok TracebackTests.test_broken_submodule) ... ok TracebackTests.test_broken_parent) ... ok TracebackTests.test_unencodable_filename) ... ok TracebackTests.test_syntax_error) ... o k TracebackTests.test_exec_failure_nested) ... ok TracebackTests.test_import_bug) ... ok TracebackTests.test_broken_parent_from) ... ok TracebackTests.test_nonexistent_module) ... ok TracebackTests.test_nonexistent_module_nested) ... ok

bedevere-bot · 2025-03-10T15:50:30Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 RHEL8 Refleaks 3.x (tier-2) has failed when building commit 2bef8ea.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/551/builds/887) and take a look at the build logs.
Check if the failure is related to this commit (2bef8ea) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/551/builds/887

Failed tests:

test_marshal
test_idle
test_import
test_datetime
test___all__
test_zipfile
test_pyclbr
test.test_pydoc.test_pydoc

Test leaking resources:

test___all__: references
test_pydoc: references
test_import: memory blocks
test_pyclbr: references
test_idle: references
test_zipfile: references
test_datetime: references
test_import: references
test_marshal: references

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_exec_failure) ... ok TracebackTests.test_syntax_error) ... ok TracebackTests.test_broken_from) ... ok TracebackTests.test_broken_submodule) ... ok TracebackTests.test_broken_parent) ... ok TracebackTests.test_unencodable_filename) ... ok TracebackTests.test_exec_failure_nested) ... ok TracebackTests.test_import_bug) ... ok TracebackTests.test_broken_parent_from) ... ok TracebackTests.test_nonexistent_module) ... ok TracebackTests.test_nonexistent_module_nested) ... ok

bedevere-bot · 2025-03-10T16:02:49Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 RHEL8 Refleaks 3.x (tier-1) has failed when building commit 2bef8ea.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/259/builds/2292) and take a look at the build logs.
Check if the failure is related to this commit (2bef8ea) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/259/builds/2292

Failed tests:

test_import

Test leaking resources:

test_xml_etree_c: references
test_inspect: references
test_ast: references
test_capi: references
test_import: memory blocks
test_pyclbr: references
test_idle: references
test_zipfile: references
test_import: references

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_exec_failure) ... ok TracebackTests.test_syntax_error) ... ok TracebackTests.test_broken_from) ... ok TracebackTests.test_broken_submodule) ... ok TracebackTests.test_broken_parent) ... ok TracebackTests.test_unencodable_filename) ... ok TracebackTests.test_exec_failure_nested) ... ok TracebackTests.test_import_bug) ... ok TracebackTests.test_broken_parent_from) ... ok TracebackTests.test_nonexistent_module) ... ok TracebackTests.test_nonexistent_module_nested) ... ok

bedevere-bot · 2025-03-10T16:27:08Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Fedora Stable Refleaks 3.x (tier-1) has failed when building commit 2bef8ea.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/320/builds/2091) and take a look at the build logs.
Check if the failure is related to this commit (2bef8ea) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/320/builds/2091

Failed tests:

test_idle
test_bdb
test_import
test_runpy
test_datetime
test___all__
test_capi
test_ast
test_decorators
test_zipfile
test_pyclbr

Test leaking resources:

test_runpy: memory blocks
test_ast: references
test___all__: references
test_runpy: references
test_capi: references
test_bdb: references
test_import: memory blocks
test_pyclbr: references
test_zipfile: references
test_idle: references
test_decorators: references
test_datetime: references
test_import: references

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_exec_failure) ... ok TracebackTests.test_syntax_error) ... ok TracebackTests.test_broken_from) ... ok TracebackTests.test_broken_submodule) ... ok TracebackTests.test_broken_parent) ... ok TracebackTests.test_unencodable_filename) ... ok TracebackTests.test_exec_failure_nested) ... ok TracebackTests.test_import_bug) ... ok TracebackTests.test_broken_parent_from) ... ok TracebackTests.test_nonexistent_module) ... ok TracebackTests.test_nonexistent_module_nested) ... ok

bedevere-bot · 2025-03-10T16:39:32Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 FreeBSD Refleaks 3.x (tier-3) has failed when building commit 2bef8ea.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1613/builds/926) and take a look at the build logs.
Check if the failure is related to this commit (2bef8ea) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1613/builds/926

Failed tests:

test_import

Test leaking resources:

test_timeit: references
test_import: memory blocks
test_idle: references
test_zipfile: references
test_import: references

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_exec_failure) ... ok TracebackTests.test_syntax_error) ... ok TracebackTests.test_broken_from) ... ok TracebackTests.test_broken_submodule) ... ok TracebackTests.test_broken_parent) ... ok TracebackTests.test_unencodable_filename) ... ok TracebackTests.test_exec_failure_nested) ... ok TracebackTests.test_import_bug) ... ok TracebackTests.test_broken_parent_from) ... ok TracebackTests.test_nonexistent_module) ... ok TracebackTests.test_nonexistent_module_nested) ... ok

bedevere-bot · 2025-03-10T18:41:23Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE RHEL8 Refleaks 3.x (tier-2) has failed when building commit 2bef8ea.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/384/builds/1779) and take a look at the build logs.
Check if the failure is related to this commit (2bef8ea) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/384/builds/1779

Failed tests:

test_ast
test_zipfile
test_idle
test_xml_etree_c
test_bdb
test_import
test_datetime

Test leaking resources:

test_xml_etree_c: references
test_ast: references
test_bdb: references
test_import: memory blocks
test_idle: references
test_zipfile: references
test_datetime: references
test_import: references

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_exec_failure) ... ok TracebackTests.test_syntax_error) ... ok TracebackTests.test_broken_from) ... ok TracebackTests.test_broken_submodule) ... ok TracebackTests.test_broken_parent) ... ok TracebackTests.test_unencodable_filename) ... ok TracebackTests.test_exec_failure_nested) ... ok TracebackTests.test_import_bug) ... ok TracebackTests.test_broken_parent_from) ... ok TracebackTests.test_nonexistent_module) ... ok TracebackTests.test_nonexistent_module_nested) ... ok

bedevere-bot · 2025-03-10T19:06:24Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64LE Fedora Stable Refleaks 3.x (tier-2) has failed when building commit 2bef8ea.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/280/builds/1711) and take a look at the build logs.
Check if the failure is related to this commit (2bef8ea) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/280/builds/1711

Failed tests:

test_idle
test_bdb
test_import
test_datetime
test_pyrepl
test_ast
test_zipfile
test_pyclbr
test.test_pydoc.test_pydoc

Test leaking resources:

test_ast: references
test_pydoc: memory blocks
test_bdb: references
test_pydoc: references
test_import: memory blocks
test_pyclbr: references
test_zipfile: references
test_idle: references
test_datetime: references
test_import: references
test_pyrepl: references

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_exec_failure) ... ok TracebackTests.test_syntax_error) ... ok TracebackTests.test_broken_from) ... ok TracebackTests.test_broken_submodule) ... ok TracebackTests.test_broken_parent) ... ok TracebackTests.test_unencodable_filename) ... ok TracebackTests.test_exec_failure_nested) ... ok TracebackTests.test_import_bug) ... ok TracebackTests.test_broken_parent_from) ... ok TracebackTests.test_nonexistent_module) ... ok TracebackTests.test_nonexistent_module_nested) ... ok

bedevere-bot · 2025-03-11T00:50:35Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows11 Refleaks 3.x (tier-1) has failed when building commit 2bef8ea.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/920/builds/1438) and take a look at the build logs.
Check if the failure is related to this commit (2bef8ea) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/920/builds/1438

Failed tests:

test_import
test_zipfile
test_multiprocessing_main_handling
test_idle

Test leaking resources:

test_import: references
test_import: memory blocks
test_zipfile: references
test_idle: references
test_multiprocessing_main_handling: memory blocks
test_multiprocessing_main_handling: references

Summary of the results of the build (if available):

==

Click to see traceback logs

TracebackTests.test_exec_failure) ... ok TracebackTests.test_syntax_error) ... ok TracebackTests.test_broken_from) ... ok TracebackTests.test_broken_submodule) ... ok TracebackTests.test_broken_parent) ... ok TracebackTests.test_unencodable_filename) ... ok TracebackTests.test_exec_failure_nested) ... ok TracebackTests.test_import_bug) ... ok TracebackTests.test_broken_parent_from) ... ok TracebackTests.test_nonexistent_module) ... ok TracebackTests.test_nonexistent_module_nested) ... ok

markshannon added 30 commits October 10, 2024 20:18

Initial implementation of tagged stackrefs for GIL build. Requires fi…
2ec65dc
…xes to immortal objects.

Merge branch 'main' into use-stackrefs
9a5c3a6

Streamline PyStackRef_XCLOSE and PyStackRef_CLEAR.
ee7f3a6

Fix error handling with not stackref outputs in code generator
21d7e87

Merge branch 'main' into use-stackrefs
370e77b

Attempt to fix _BINARY_OP_INPLACE_ADD_UNICODE and use tag references …
2636c04
…in LOAD_CONST

Merge branch 'main' into use-stackrefs
5a38e03

Show frame locals in lltrace
a79aa47

Fix _PyFrame_Copy and add some asserts to make reference heap safe
a90e644

Improve efficiency of _PyStackRef_FromPyObjectNew a bit
83e0323

Merge branch 'main' into use-stackrefs
7fb6a48

Merge branch 'main' into use-stackrefs
7e6deef

Make making heap safe more efficient
f03f745

Merge branch 'main' into use-stackrefs
67670f0

Fix up after merge
ff9d044

Merge branch 'main' into use-stackrefs
3334c14

Restore immortality bit
b7a8b5d

Fix validity check
49d0d42

Add PyStackRef_IsMortal
3a12d18

Reduce overhead of stackrefs a tiny bit for mortal objects
497fb9f

Merge branch 'main' into use-stackrefs
4768a38

Attempt to use tagged RC for a bit of extra speed
806c28a

Merge branch 'main' into use-stackrefs
c540f28

Revert addition of LOAD_FAST_WITH_COUNT
0c20416

Avoid INCREF + Steal pairs by using New or return stack ref directly
c88bcbc

Avoid masking when comparing to True and False
f5dec96

Merge branch 'main' into use-stackrefs
bf2dca5

Revert not-quite-true assert
1069d98

Merge branch 'main' into use-stackrefs
2e752fc

Fix up after merge
d23ae47

markshannonand others added 2 commits March 3, 2025 15:12

Merge branch 'main' into use-stackrefs
3ef8571

Update Misc/NEWS.d/next/Core_and_Builtins/2025-02-17-12-48-00.gh-issu…
e2f1387
…e-127705.Qad2hx.rst Co-authored-by: Irit Katriel <1055913+iritkatriel@users.noreply.github.com>

mpage requested a review from colesbury March 3, 2025 17:47

ericsnowcurrently reviewed Mar 4, 2025
View reviewed changes

Include/internal/pycore_frame.hShow resolvedHide resolved

iritkatriel reviewed Mar 5, 2025
View reviewed changes

Include/internal/pycore_stackref.hShow resolvedHide resolved

Use consistent return types for PyStackRef_IsHeapSafe
36f6034

iritkatriel approved these changes Mar 5, 2025
View reviewed changes

bedevere-appbot added awaiting merge and removed awaiting core review labels Mar 5, 2025

markshannon added 2 commits March 7, 2025 14:54

Drop 'immortal' bit. Only use 'counted' bit.
0f9c865

Remove use of Py_TAG_IMMORTAL
37f1e18

Fidget-Spinner reviewed Mar 9, 2025
View reviewed changes

Rename functions and add comment
d744179

iritkatriel approved these changes Mar 10, 2025
View reviewed changes

markshannon merged commit 2bef8ea into python:mainMar 10, 2025
71 checks passed

bedevere-appbot removed the awaiting merge label Mar 10, 2025

markshannon mentioned this pull request Mar 11, 2025
GH-127705: Check for immortality in refcount accounting #131072
Merged

markshannon deleted the use-stackrefs branch March 11, 2025 11:53

colesbury mentioned this pull request Dec 9, 2025
Merge _PyStackRef definitions for Py_GIL_DISABLED and default builds #142472
Closed

Uh oh!

GH-127705: Use _PyStackRefs in the default build.#127875

GH-127705: Use _PyStackRefs in the default build. #127875

Uh oh!

Conversation

markshannon commented Dec 12, 2024• edited by bedevere-app botLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ericsnowcurrently left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mpage commented Mar 6, 2025

Uh oh!

markshannon commented Mar 7, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fidget-SpinnerMar 9, 2025

Choose a reason for hiding this comment

Uh oh!

markshannonMar 10, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bedevere-bot commented Mar 10, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Mar 10, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Mar 10, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Mar 10, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Mar 10, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Mar 10, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Mar 10, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Mar 10, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

bedevere-bot commented Mar 11, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

GH-127705: Use `_PyStackRef`s in the default build.#127875

GH-127705: Use `_PyStackRef`s in the default build. #127875

markshannon commented Dec 12, 2024•
edited by bedevere-app bot
Loading

markshannon commented Mar 7, 2025•
edited
Loading

markshannonMar 10, 2025•
edited
Loading