GH-96764: rewrite `asyncio.wait_for` to use `asyncio.timeout`#98518

kumaraditya303 · 2022-10-21T12:59:07Z

This PR changes asyncio.wait_for to use asyncio.timeout as its underlying implementation. It simplifies the code and makes it easy to understand the cancellation semantics as both asyncio.timeout and asyncio.wait_for behaves similarly.

Fixes#86296
Fixes#81839
Fixes#96764

Issue: Rewrite asyncio.wait_for using asyncio.timeout #96764

kumaraditya303 · 2022-10-21T18:09:51Z

This is more of a POC at this point. I think it is worth changing wait_for to use timeout in 3.12+. This seems to fix some of the open bugs of wait_for, I'll add those tests to make sure.

cc @gvanrossum

gvanrossum

You got a bit farther than I did when I tried this, but I'm still concerned why you had to modify several tests. (Also, there's one test that fails in CI, but doesn't appear to fail when I run it locally.)

Lib/asyncio/tasks.py

Lib/test/test_asyncio/test_futures2.py

Lib/test/test_asyncio/test_waitfor.py

Lib/asyncio/tasks.py

Misc/NEWS.d/next/Library/2022-10-22-09-26-43.gh-issue-96764.Dh9Y5L.rst

Lib/test/test_asyncio/test_waitfor.py

gvanrossum

LG (one naming nit).

Lib/test/test_asyncio/test_waitfor.py

gvanrossum · 2022-10-25T18:34:13Z

I had hoped to first merge one of the alternatives that can be backported to 3.10, so we can declare this fixed in 3.10. Then we would merge this one on top of that, but only in 3.11 and main.

But we could also just do a custom fix for 3.10 based on one of @twisteroidambassador's PRs. (I wish you had linked this PR to the same issue rather than creating a new issue, since it's all related.)

gvanrossum · 2022-10-25T18:36:20Z

In particular, @twisteroidambassador has this PR: #98607

Lib/test/test_asyncio/test_futures2.py

a new task so no sleep is required

Fallback to using async_timeout on older python asyncio.wait_for has some underlying problems that are only fixed in cpython 3.12. See python/cpython#98518

asyncio.wait_for has some underlying problems that are only fixed in cpython 3.12. Use async_timeout instead until the minimum supported version is 3.11+ and it can be replaced with asyncio.timeout See python/cpython#98518

async_timeout does not suffer from the same race problems as asyncio.wait_for see python/cpython#98518 for more details

wasted some time because asyncio.wait_for() was suppressing cancellations. [0][1][2] deja vu... [3] Looks like this is finally getting fixed in cpython 3.12 [4] So far away... In attempt to avoid encountering this again, let's try using asyncio.timeout in 3.11, which is how upstream reimplemented wait_for in 3.12 [4], and aiorpcx.timeout_after in 3.8-3.10. [0] python/cpython#86296 [1] https://bugs.python.org/issue42130 [2] https://bugs.python.org/issue45098 [3] kyuupichan/aiorpcX#44 [4] python/cpython#98518

`asyncio.wait_for` creates another tasks which leads to some race conditions in cancelation and a performance hit cpython 3.12 will change the underlying implementation of `asyncio.wait_for` to use `asyncio.wait` but that is still a long way off for many people: python/cpython#98518

* Switch usage of asyncio.wait_for to async_timeout `asyncio.wait_for` creates another tasks which leads to some race conditions in cancelation and a performance hit cpython 3.12 will change the underlying implementation of `asyncio.wait_for` to use `asyncio.wait` but that is still a long way off for many people: python/cpython#98518 * adjust ci

This is an implementation of Python's `asyncio.timeout()` which was introduced in 3.11. We strongly encourage the use of the following approach instead of `asyncio.wait_for()`. async with timeout(delay): # Your async code In Python 3.12, `asyncio.wait_for()` is implemented in terms of `asyncio.timeout()` which fixed a bunch of issues. However, this was never backported (because of the lack of `async.timeout()`) and there are still many remainig issues, specially in Python 3.10, in `async.wait_for()`. See python/cpython#98518

This patch implements `asyncio.timeout()` which was introduced in 3.11, as well +as `asyncio.wait_for()` on top of it. In Python 3.12, `asyncio.wait_for()` is implemented in terms of `asyncio.timeout()` which fixed a bunch of issues. However, this was never backported (because of the lack of `async.timeout()`) and there are still many remainig issues, specially in Python 3.10, in `async.wait_for()`. See python/cpython#98518

This patch implements a custom `asyncio.wait_for()` on top of an also custom `asyncio.timeout()`. `asyncio.timeout()` which was introduced in 3.11. In Python 3.12, `asyncio.wait_for()` is implemented in terms of `asyncio.timeout()` which fixed a bunch of issues. However, this was never backported (because of the lack of `async.timeout()`) and there are still many remainig issues, specially in Python 3.10, in `async.wait_for()`. See python/cpython#98518

This patch uses `wait_for2` package to implement `asyncio.wait_for()` for Python < 3.12. In Python 3.12, `asyncio.wait_for()` is implemented in terms of `asyncio.timeout()` which fixed a bunch of issues. However, this was never backported (because of the lack of `async.timeout()`) and there are still many remainig issues, specially in Python 3.10, in `async.wait_for()`. See python/cpython#98518

bedevere-bot added the awaiting review label Oct 21, 2022

kumaraditya303 force-pushed the waitfor branch 2 times, most recently from 45d0725 to 6c944d9Compare October 21, 2022 13:09

gvanrossum reviewed Oct 21, 2022
View reviewed changes

Lib/asyncio/tasks.pyShow resolvedHide resolved
Lib/test/test_asyncio/test_futures2.pyShow resolvedHide resolved
Lib/test/test_asyncio/test_waitfor.pyShow resolvedHide resolved

gvanrossum reviewed Oct 21, 2022
View reviewed changes

Lib/asyncio/tasks.pyShow resolvedHide resolved

gvanrossum mentioned this pull request Oct 21, 2022
AsyncIO's wait_for can hide cancellation in a rare race condition #86296
Closed

gvanrossum reviewed Oct 22, 2022
View reviewed changes

Misc/NEWS.d/next/Library/2022-10-22-09-26-43.gh-issue-96764.Dh9Y5L.rst Outdated Show resolvedHide resolved

gvanrossum reviewed Oct 22, 2022
View reviewed changes

Lib/test/test_asyncio/test_waitfor.pyShow resolvedHide resolved

gvanrossum approved these changes Oct 24, 2022
View reviewed changes

Lib/test/test_asyncio/test_waitfor.py Outdated Show resolvedHide resolved

bedevere-bot added awaiting merge and removed awaiting review labels Oct 24, 2022

kumaraditya303 marked this pull request as ready for review October 24, 2022 06:03

kumaraditya303 requested review from 1st1 and asvetlov as code owners October 24, 2022 06:03

kumaraditya303 added type-feature A feature request or enhancement topic-asyncio labels Oct 25, 2022

kumaraditya303 requested a review from gvanrossum October 25, 2022 17:40

akulakov reviewed Nov 3, 2022
View reviewed changes

Lib/test/test_asyncio/test_futures2.py Outdated Show resolvedHide resolved

bedevere-bot mentioned this pull request Nov 30, 2022
Rewrite asyncio.wait_for using asyncio.timeout#96764
Closed

robsdedude mentioned this pull request Dec 14, 2022
GH-86296: Fix for asyncio.wait_for() swallowing cancellation, and add tests #98607
Closed

python deleted a comment from netlifybotDec 14, 2022

kumaraditya303 added 6 commits January 5, 2023 15:24

rewrite wait_for to use timeouts.timeout
487eac1

skip one test
794cb3a

fix test as wait_for no longer creates
13ab78f
a new task so no sleep is required

remove outdated tests
1eb0053

fix future test
c704996

more tests
bdaed9d

This was referenced Apr 10, 2023
Blacken code to fix lint CI aiolifx/aiolifx#73
Merged
Suppress CancelledError in zwave-js unload home-assistant/core#91222
Merged

farmio mentioned this pull request Apr 13, 2023
Use asyncio.timeout instead of asyncio.wait_forXKNX/xknx#1246
Merged
7 tasks

willingc mentioned this pull request Apr 26, 2023
GH-98388: Add tests for happy eyeballs and its internal workings #98389
Closed

bdraco mentioned this pull request Jul 21, 2023
Replace asyncio.wait_for with async-timeout python-kasa/python-kasa#480
Merged

bdraco added a commit to bdraco/ha-HAP-python that referenced this pull request Jul 23, 2023
Switch timeouts to use async_timeout
b1e0a11
async_timeout does not suffer from the same race problems as asyncio.wait_for see python/cpython#98518 for more details

bdraco mentioned this pull request Jul 23, 2023
Switch timeouts to use async_timeout ikalchev/HAP-python#447
Merged

bdraco mentioned this pull request Aug 31, 2023
Switch usage of asyncio.wait_for to async_timeout JeffLIrion/python-androidtv#337
Merged

cdce8p mentioned this pull request Sep 18, 2023
TEST: Add 3.12-dev [ci] cdce8p/ha-core#53
Closed
32 tasks

cdce8p mentioned this pull request Sep 27, 2023
Fix additional test cases for Python 3.12 home-assistant/core#101006
Merged
20 tasks

bdraco mentioned this pull request Nov 8, 2023
Make check_port an async function home-assistant/supervisor#4677
Merged
12 tasks

aconchillo mentioned this pull request Aug 20, 2025
implement custom asyncio.wait_for() pipecat-ai/pipecat#2476
Merged

skshetry mentioned this pull request Oct 30, 2025
Python3 runtime often stalls in _finalize() or close() when garbage collecting fsspec/sshfs#63
Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

GH-96764: rewrite `asyncio.wait_for` to use `asyncio.timeout`#98518

GH-96764: rewrite `asyncio.wait_for` to use `asyncio.timeout`#98518

Uh oh!

kumaraditya303 commented Oct 21, 2022•
edited
Loading

Uh oh!

kumaraditya303 commented Oct 21, 2022

Uh oh!

gvanrossum left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gvanrossum left a comment

Uh oh!

Uh oh!

gvanrossum commented Oct 25, 2022

Uh oh!

gvanrossum commented Oct 25, 2022

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

GH-96764: rewrite asyncio.wait_for to use asyncio.timeout#98518

GH-96764: rewrite asyncio.wait_for to use asyncio.timeout#98518

Uh oh!

Conversation

kumaraditya303 commented Oct 21, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kumaraditya303 commented Oct 21, 2022

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gvanrossum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gvanrossum commented Oct 25, 2022

Uh oh!

gvanrossum commented Oct 25, 2022

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

GH-96764: rewrite `asyncio.wait_for` to use `asyncio.timeout`#98518

GH-96764: rewrite `asyncio.wait_for` to use `asyncio.timeout`#98518

kumaraditya303 commented Oct 21, 2022•
edited
Loading