Async Call-Stack Reconstruction

[mpage]

BPO	46892
Nosy	@gvanrossum, @DinoV, @asvetlov, @1st1, @kumaraditya303, @itamaro, @mpage

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-03-01.22:13:38.844>
labels = ['interpreter-core', '3.11', 'type-feature', 'expert-asyncio']
title = 'Async Call-Stack Reconstruction'
updated_at = <Date 2022-03-12.09:30:23.838>
user = 'https://github.com/mpage'

bugs.python.org fields:

activity = <Date 2022-03-12.09:30:23.838>
actor = 'asvetlov'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'asyncio']
creation = <Date 2022-03-01.22:13:38.844>
creator = 'mpage'
dependencies = []
files = []
hgrepos = []
issue_num = 46892
keywords = []
message_count = 5.0
messages = ['414301', '414884', '414930', '414948', '414985']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'dino.viehland', 'asvetlov', 'yselivanov', 'kumaraditya', 'itamaro', 'mpage']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46892'
versions = ['Python 3.11']

Linked PRs

• gh-91048: Add support for reconstructing async call stacks #103976
• GH-91048: Add utils for capturing async call stack for asyncio programs and enable profiling #124640
• GH-91048: Don't attempt to run on FreeBSD #129189
• gh-91048: Also clear and set ts->asyncio_running_task with eager tasks #129197
• gh-91048: Improve example in asyncio-graph.rst doc #129224
• gh-91048: fix thread safety for asyncio stack introspection APIs #129399
• GH-91048: Correct error path in testexternalinspection #129557
• gh-91048: fix _asyncio.future_discard_from_awaited_by #129731
• gh-91048: Add ability to list all pending asyncio tasks in a process remotely #132807
• gh-91048: Refactor _testexternalinspection and add Windows support #132852
• gh-91048: Fix _testexternalinspection.c on FreeBSD #132945
• gh-91048: Prevent optimizing away the asyncio debug offsets structure on Windows #132963
• gh-91048: Chain some exceptions in _testexternalinspection.c #132970
• GH-91048: Add utils for printing the call stack for asyncio tasks #133284
• gh-133141: Relax test_async_global_awaited_by to fix flakyness #133368
• gh-91048: Add filename and line number to external inspection routines #133385
• GH-91048: Minor fixes for _remotedebugging & rename to _remote_debugging #133398
• gh-91048: Add filename:line_no information to asyncio pstree #133478
• gh-91048: Fix error path result in _remote_debugging_module #134347
• [3.14] gh-91048: Fix error path result in _remote_debugging_module (GH-134347) #134399
• gh-91048: Refactor and optimize remote debugging module #134652
• [3.14] gh-91048: Refactor and optimize remote debugging module (#134652) #134673
• gh-91048: Add better error messages for remote debugging for CI builds #134682
• gh-91048: Guard against TARGET_OS_OSX for compat with simulators in remote_debug.h #134711
• gh-91048: Correct Apple platform includes for iOS #134712
• [3.14] gh-91048: Correct Apple platform includes for iOS. (GH-134712) #134714
• [3.14] gh-91048: Add better error messages for remote debugging for CI builds (GH-134682) #134719
• gh-91048: Reorder result tuple of parse_code_object #134898
• [3.14] gh-91048: Reorder result tuple of parse_code_object (GH-134898) #134956
• gh-91048: Reorder result tuple in asyncio/tools.py #134957
• gh-91048: Fix external inspection multi-threaded performance #136005
• [3.14] gh-91048: Fix external inspection multi-threaded performance (GH-136005) #136080
• gh-91048: Revert the memory cache removal for remote debugging #136440
• [3.14] gh-91048: Revert the memory cache removal for remote debugging (GH-136440) #136443
• gh-91048: Refactor common data into context object in Modukes/_remote_debugging #142879

[mpage]

Profiling tools that use the call-stack (i.e. all of them) paint an incomplete picture of what’s really going on in async-heavy codebases. They can only show the stack of the currently executing task; they miss the chain of awaitables that are transitively waiting on the current task. To remedy this, we have added support in Cinder to expose the async call-stack. This consists of the call stack for the currently executing task, followed by the chain of awaitables that are transitively reachable from the currently executing task. See below for a clarifying example.

async def f1():
 return await f2()

async def f2():
 return await asyncio.ensure_future(f3())

async def f3():
 return await f4()

async def f4():
 await asyncio.sleep(10)
 return 42

When retrieved from f4, the two different stacks (top-of-stack last) are:
sync - [f3, f4]
async - [f1, f2, f3, f4]

We’d like to merge our implementation into CPython so that other heavy users of asyncio can benefit. This will consist of a few parts:

1. A new limited C-API to set and retrieve the “awaiter” of an awaitable object.
2. Additions to PyAsyncMethods to store function pointers for setting and retrieving the awaiter on instances.
3. An API in managed code to retrieve the async call stack as a list of fully qualified names (i.e. <module>:<class>.<function>).

[asvetlov]

We’d like to merge our implementation into CPython

Could you provide a link first, please?

[mpage]

Sorry for the confusion, I'm working on a PR. I filed the BPO to gauge interest in the feature.

[gvanrossum]

I've recently dabbled a bit in some new primitives for asyncio, and based on that experience I think this would be very useful.

IIRC Trio does this (presumably at considerable cost) in userland.

[asvetlov]

The idea looks interesting.
The devil in the details I guess.
I'm curious what is the memory and performance penalty.
Waiting for the PR as the discussion starting point.