Change create_task type

[sobolevn]

I've manually tested that Awaitables are not supported:

importasyncioclassMy: def__await__(self): asyncdefinner(): print('here') returninnerasyncdefmain(): awaitasyncio.create_task(My()) asyncio.run(main())

Output:

Traceback (most recent call last): File "/Users/sobolev/Desktop/cpython/build/ex.py", line 15, in <module> asyncio.run(main()) File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete return future.result() File "/Users/sobolev/Desktop/cpython/build/ex.py", line 13, in main await asyncio.create_task(My()) File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/tasks.py", line 361, in create_task task = loop.create_task(coro) File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/base_events.py", line 433, in create_task task = tasks.Task(coro, loop=self, name=name) TypeError: a coroutine was expected, got <__main__.My object at 0x10c38fe20> 

But, Generator objects are also supported:

importasynciodefgen(): i=yieldprint(1, i) # prints: `1 None`asyncdefmain(): awaitasyncio.create_task(gen()) asyncio.run(main())

Closes#6770

[sobolevn]

Notice that _TaskYieldType was not used.

[Akuli]

This doesn't seem right. Event loops typically use yields to communicate between tasks and the loop, so the generator must yield objects of a specific type.

[sobolevn]

Some experiments:

importasynciodefgen(): i=yieldprint(1, i) # prints: `1, None`return'a'asyncdefmain(): print('res', awaitasyncio.create_task(gen())) # prints: `res, a`asyncio.run(main())

And:

importasynciodefgen(): i=yieldprint(1, i) # prints: `1, None`yield'a'asyncdefmain(): print('res', awaitasyncio.create_task(gen())) asyncio.run(main())

1 None Traceback (most recent call last): File "/Users/sobolev/Desktop/cpython/build/ex.py", line 10, in <module> asyncio.run(main()) File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/Users/sobolev/.pyenv/versions/3.9.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete return future.result() File "/Users/sobolev/Desktop/cpython/build/ex.py", line 8, in main print('res', await asyncio.create_task(gen())) # prints: `res, a` File "/Users/sobolev/Desktop/cpython/build/ex.py", line 5, in gen yield 'a' RuntimeError: Task got bad yield: 'a' 

[sobolevn]

Ok, we have some problems:

self._navigationPromise=self._loop.create_task(asyncio.wait([ self._lifecycleCompletePromise, self._createTimeoutPromise(), ], return_when=concurrent.futures.FIRST_COMPLETED))

Here asyncio.wait returns Future, but now self._loop.create_task() wants Coroutine. Future is not compatible with Coroutine. 🤔

[Akuli]

Doesn't look like a Future to me:

Python 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import asyncio >>> asyncio.wait([]) <coroutine object wait at 0x7f90e8cef140> 

[sobolevn]

@Akuli sorry, I had to clarify that I'm talking about typeshed's definition:

typeshed/stdlib/asyncio/tasks.pyi

Lines 235 to 239 in 505ea72

	defwait(fs: Iterable[_FT], *, timeout: float|None= ..., return_when: str= ...) ->Future[tuple[set[_FT], set[_FT]]]: ... # type: ignore[misc]
	@overload
	defwait(
	fs: Iterable[Awaitable[_T]], *, timeout: float|None= ..., return_when: str= ...
	) ->Future[tuple[set[Task[_T]], set[Task[_T]]]]: ...

[Akuli]

A few observations:

• The coro argument gets passed to Task.__init__, so that should be changed too.
• Task.__init__ uses asyncio.coroutines.iscoroutine, which is an optimized isinstance(obj, asyncio.coroutines._COROUTINE_TYPES), not to be confused with inspect.iscoroutine which is isinstance(obj, types.CoroutineType).
• asyncio is a mess :(

[sobolevn]

Ok, I have no idea what is going on:

• Why some definitions are async def, while others are sync def that return Future?
• Why do they return Future, why not Coroutine?
• Why stubtest allows this?

[sobolevn]

Ok, my plan is to sync all async def in source code and async def in typeshed (unless there are some problems that I don't know about).

[hauntsaninja]

Probably shouldn't merge this as is, given the false positives. The one in my code could be potentially worked around by changing AsyncIterator.__anext__ to return a Coroutine.

Re stubtest:
It's one of many tricks stubtest doesn't know about / mypy itself largely treats def f() -> Coroutine[Any, Any, int] the same as async def f() -> int so it's not something you're forced to handle. Can't remember if I tried adding support for this to stubtest; there were several things I chose not to add based on typeshed error counts. Might have been affected by older Pythons, e.g. on Python 3.5 several functions are @coroutine not async def.

[sobolevn]

Totally do not merge this as-is, I will spend several days fixing all the async def / def stuff. And hopefully after that we can return to this issue.

[srittau]

I've converted this to a draft for now. Please mark as "ready for review" when it is. :)

[sobolevn]

I've changed all things I could find to be async def:

typeshed/stdlib/asyncio/tasks.pyi

Lines 235 to 239 in 7bc9c16

	asyncdefwait(fs: Iterable[_FT], *, timeout: float|None= ..., return_when: str= ...) ->tuple[set[_FT], set[_FT]]: ... # type: ignore[misc]
	@overload
	asyncdefwait(
	fs: Iterable[Awaitable[_T]], *, timeout: float|None= ..., return_when: str= ...
	) ->tuple[set[Task[_T]], set[Task[_T]]]: ...

But, mypy-primer found some new ones 🤔

[sobolevn]

Analysis:

• homeasistant annotation is not correct, it uses Awaitable, which is not supported by create_task: https://github.com/home-assistant/core/blob/19b7454161046d68dc388e7ff1ffdd0573855449/homeassistant/core.py#L439 I will send a PR with the fix to their repo 👍
• @hauntsaninja, what do you think about https://github.com/hauntsaninja/boostedblob/blob/master/boostedblob/boost.py#L547 ?
• Argument 1 to "create_task" has incompatible type "Awaitable[Sequence[Any]]"; expected "Union[Generator[Any, None, <nothing>], Coroutine[Any, Any, <nothing>]]" seems correct to me

This error does not look good to me: https://github.com/Yelp/paasta/blob/5e504597ea5a2af952a12785808e523b53b7c01d/paasta_tools/instance/kubernetes.py#L545 🤔

Aha! Found the reason: the decorator annotation is not quite correct: https://github.com/Yelp/paasta/blob/5e504597ea5a2af952a12785808e523b53b7c01d/paasta_tools/async_utils.py#L93-L97

[srittau]

@sobolevn If I understand correctly, in your opinion the primer diffs all show genuine problems?

[sobolevn]

@srittau yes, I think so! I've opened #7105 to fix 2.

[srittau]

Ok, I'll have a look once the merge conflicts are resolved.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

core (https://github.com/home-assistant/core) + homeassistant/core.py:439: error: Argument 1 to "create_task" of "AbstractEventLoop" has incompatible type "Awaitable[_R]"; expected "Union[Coroutine[Any, Any, _R], Generator[Any, None, _R]]" [arg-type]+ homeassistant/core.py:470: error: Need type annotation for "task" [var-annotated]+ homeassistant/core.py:470: error: Argument 1 to "create_task" of "AbstractEventLoop" has incompatible type "Awaitable[_R]"; expected "Union[Coroutine[Any, Any, <nothing>], Generator[Any, None, <nothing>]]" [arg-type] paasta (https://github.com/yelp/paasta) - paasta_tools/instance/kubernetes.py:553: error: Argument 1 to "append" of "list" has incompatible type "Task[Sequence[Any]]"; expected "Future[Dict[str, Any]]"+ paasta_tools/instance/kubernetes.py:545: error: Need type annotation for "pods_task"+ paasta_tools/instance/kubernetes.py:546: error: Argument 1 to "create_task" has incompatible type "Awaitable[Sequence[Any]]"; expected "Union[Generator[Any, None, <nothing>], Coroutine[Any, Any, <nothing>]]"+ paasta_tools/instance/kubernetes.py:1017: error: Need type annotation for "pods_task"+ paasta_tools/instance/kubernetes.py:1018: error: Argument 1 to "create_task" has incompatible type "Awaitable[Sequence[Any]]"; expected "Union[Generator[Any, None, <nothing>], Coroutine[Any, Any, <nothing>]]"+ paasta_tools/instance/kubernetes.py:1202: error: Need type annotation for "pods_task"+ paasta_tools/instance/kubernetes.py:1203: error: Argument 1 to "create_task" has incompatible type "Awaitable[Sequence[Any]]"; expected "Union[Generator[Any, None, <nothing>], Coroutine[Any, Any, <nothing>]]"

[srittau]

Nit for future reference: Coroutine should now be imported from collections.abc.

[sobolevn]

Thanks everyone 🎉