`IsolatedAsyncioTestCase` does not call `asyncio.set_event_loop` before `setUp` anymore, `asyncio.Runner`+`PidFdChildWatcher` leaves zombie processes

[graingert]

demo reproducer:

import unittest
import asyncio

class DemoTestCase1(unittest.IsolatedAsyncioTestCase):
 def setUp(self):
 self.loop = asyncio.get_event_loop_policy().get_event_loop()

 async def test_demo(self):
 print("hello!")


class DemoTestCase2(unittest.IsolatedAsyncioTestCase):
 def setUp(self):
 self.loop = asyncio.get_event_loop_policy().get_event_loop()

 async def test_demo(self):
 print("hello!")

if __name__ == "__main__":
 unittest.main()

see also #93896

 graingert@conscientious  testing310  ~/projects  python3.10 foo.py 
hello!
.hello!
.
----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK

 graingert@conscientious  testing310  ~/projects  python3.11 foo.py 
hello!
.E
======================================================================
ERROR: test_demo (__main__.DemoTestCase2.test_demo)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/lib/python3.11/unittest/async_case.py", line 82, in _callSetUp
 self._asyncioTestContext.run(self.setUp)
 File "/home/graingert/projects/foo.py", line 14, in setUp
 self.loop = asyncio.get_event_loop_policy().get_event_loop()
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/lib/python3.11/asyncio/events.py", line 677, in get_event_loop
 raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'MainThread'.

----------------------------------------------------------------------
Ran 2 tests in 0.002s

FAILED (errors=1)

it looks like this is because asyncio.Runner calls asyncio.set_event_loop( in Runner.run

[graingert]

it looks like asyncio.set_event_loop is also being called an additional time per call to asyncio.Runner.run which will call get_child_watcher().attach_loop(loop) every call which causes the PidfdChildWatcher to blow away its callbacks

[graingert]

yep that's exactly what's happening here:

import asyncio
import sys

asyncio.set_child_watcher(asyncio.PidfdChildWatcher())

tasks = set()
pids = set()


async def subprocess():
 proc = await asyncio.create_subprocess_exec(
 sys.executable,
 "-c",
 "import time; print('hello!'); import time; time.sleep(2); print('goodbye')",
 )
 pids.add(proc.pid)
 print(await proc.communicate())


async def main():
 task = asyncio.create_task(subprocess())
 tasks.add(task)
 await asyncio.sleep(1)


async def main2():
 try:
 async with asyncio.timeout(3):
 await asyncio.gather(*tasks)
 except TimeoutError:
 import psutil
 for pid in pids:
 print(psutil.Process(pid))
 else:
 print("success!")

if __name__ == "__main__":
 with asyncio.Runner() as runner:
 runner.run(main())
 runner.run(main2())

output

hello!
goodbye
psutil.Process(pid=406749, name='python', status='zombie', started='10:13:40')

[graingert]

#95584 is vaguely related

[graingert]

The issue here is that asyncio.Runner calls set_event_loop too late and too often

In addition child watchers that use attach_loop are prone to unusual behaviour when the main thread switches loop

[graingert]

It looks like IsolatedAsyncioTestCase when the Pidfdchildwatcher is used will break a subprocess started in asyncSetUp