GH-120024: Use pointer for stack pointer#121923
Merged
Uh oh!
There was an error while loading. Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Fidget-Spinner approved these changes Jul 17, 2024
Member
Fidget-Spinner left a comment
Fidget-Spinner left a comment There was a problem hiding this comment.
Even without deferred refcounts, this is a simplification in my eyes (and also brings Tier 1 and Tier 2 representations closer together), so we should do this regardless.
taegyunkim added a commit to DataDog/dd-trace-py that referenced this pull request Dec 11, 2025
## Description This PR adds support for Python 3.14 in the profiler by updating it to handle CPython internal changes. ### Key CPython changes addressed **`_PyInterpreterFrame` Structure Changes** 1. Moved from `Include/internal/pycore_frame.h` to `Include/internal/pycore_interpframe_structs.h` 2. `PyObject *f_executable` and `PyObject *f_funcobj` changed to `_PyStackRef` type. Profilers like us now need to clear the LSB of these fields to get the `PyObject*`. See python/cpython#123923 for details 3. `int stacktop` field removed, replaced with `_PyStackRef *stackpointer` pointer. See python/cpython#121923 (GH-120024) for details 4. `PyObject *localsplus[1]` changed to `_PyStackRef localsplus[1]`. See python/cpython#118450 (gh-117139) for details **`FutureObj`/`TaskObj` Changes** 1. Added fields: `awaited_by`, `is_task`, `awaited_by_is_set` in `FutureObj_HEAD` macro 2. Added `struct llist_node_task_node` field for linked-list storage **Asyncio Task Storage Changes** Prior to Python 3.14, - All tasks are stored in `_scheduled_tasks` WeakSet ([exported](https://github.com/python/cpython/blob/e96367da1fdc1e1cf17ca523e93a127b1961b443/Modules/_asynciomodule.c#L3738) from C extension) - Eager tasks are stored in `_eager_tasks` set ([exported](https://github.com/python/cpython/blob/e96367da1fdc1e1cf17ca523e93a127b1961b443/Modules/_asynciomodule.c#L3742) from C extension) From Python 3.14, - Native `asyncio.Tasks` are stored in a linked-list (`struct llist_node`) per thread and per interpreter - [Per-thread](https://github.com/python/cpython/blob/0114178911f8713bfcb935ff5542fe61b4a5d551/Include/internal/pycore_tstate.h#L46): `tstate->asyncio_tasks_head` (in `_PyThreadStateImpl`) - [Per-interpreter](https://github.com/python/cpython/blob/0114178911f8713bfcb935ff5542fe61b4a5d551/Include/internal/pycore_interp_structs.h#L897): `interp->asyncio_tasks_head` (for lingering tasks) - Each `TaskObj` has a `task_node` field with `next` and `prev` pointers - Third-party tasks: Still stored in Python-level `_scheduled_tasks` WeakSet (now Python-only, not exported from C extension) - Eager tasks: Still stored in Python-level `_eager_tasks` set ### Implementation Summary - **Frame reading** (`frame.h`, `frame.cc`): Updated header includes to use `pycore_interpframe_structs.h` for Python 3.14+. Implemented tagged pointer handling: clear LSB of `f_executable` to recover `PyObject*` (per gh-123923). Replaced `stacktop` field access with `stackpointer` pointer arithmetic for stack depth calculation. Updated `PyGen_yf()` to use `_PyStackRef` and `stackpointer[-1]` instead of `localsplus[stacktop-1]`. Added handling for `FRAME_OWNED_BY_INTERPRETER` frame type (introduced in 3.14). - **Task structures** (`cpython/tasks.h`): Added Python 3.14+ `FutureObj_HEAD` macro with new fields: `awaited_by`, `is_task`, `awaited_by_is_set`. Added `struct llist_node task_node` field to `TaskObj` for linked-list storage. Updated `PyGen_yf()` implementation to handle `_PyStackRef` and `stackpointer` instead of `stacktop`. - **Asyncio discovery** (`tasks.h`, `threads.h`): Implemented `get_tasks_from_linked_list()` to safely iterate over circular linked-lists with iteration limits (`MAX_ITERATIONS = 2 << 15`). Added `get_tasks_from_thread_linked_list()` to read tasks from `_PyThreadStateImpl.asyncio_tasks_head` (per-thread active tasks). Added `get_tasks_from_interpreter_linked_list()` to read lingering tasks from `PyInterpreterState.asyncio_tasks_head` (per-interpreter). Updated `get_all_tasks()` to handle both linked-list (native `asyncio.Task` instances) and WeakSet (third-party tasks). - **Python integration** (`_asyncio.py`): Added compatibility handling for `BaseDefaultEventLoopPolicy` → `_BaseDefaultEventLoopPolicy` rename in 3.14. Updated `_scheduled_tasks` access to handle Python-only WeakSet (no longer exported from C extension in 3.14+). ## Testing All existing tests pass except for tests/profiling/collector/test_memalloc.py which needed some edits. ## Risks <!-- Note any risks associated with this change, or "None" if no risks --> ## Additional Notes --------- Co-authored-by: Brett Langdon <brett.langdon@datadoghq.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To support deferred reference counting, the stack pointer will need to be spilled around many calls.
The current implementation of the stack pointer as an offset from the 0th local will be too slow for that as each spill and reload will need to perform a calculation of the stack pointer from the stacktop or vice-versa.
This PR changes the
stacktopoffset tostackpointerpointer, so that no calculation is needed for the spill.