Borrow asyncio ssl implementation from uvloop

[asvetlov]

BPO	44011
Nosy	@asvetlov, @1st1, @pablogsal, @kumaraditya303
PRs	bpo-44011: New asyncio ssl implementation #17975 bpo-44011: Fix asyncio tests without ssl module (GH-25840) #25840 bpo-44011: Increase test timeout #25842 bpo-44011: Increase timeouts for asyncio SSL tests #25846 bpo-44011: Revert "New asyncio ssl implementation (GH-17975)" #25848 bpo-44011: New asyncio ssl implementation #31275 bpo-44011: add docs for ssl_shutdown_timeout parameter #31597

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/asvetlov'
closed_at = <Date 2022-02-22.21:12:45.984>
created_at = <Date 2021-05-02.20:33:39.996>
labels = ['3.11', 'type-feature', 'expert-asyncio']
title = 'Borrow asyncio ssl implementation from uvloop'
updated_at = <Date 2022-02-26.13:06:50.530>
user = 'https://github.com/asvetlov'

bugs.python.org fields:

activity = <Date 2022-02-26.13:06:50.530>
actor = 'asvetlov'
assignee = 'asvetlov'
closed = True
closed_date = <Date 2022-02-22.21:12:45.984>
closer = 'asvetlov'
components = ['asyncio']
creation = <Date 2021-05-02.20:33:39.996>
creator = 'asvetlov'
dependencies = []
files = []
hgrepos = []
issue_num = 44011
keywords = ['patch']
message_count = 18.0
messages = ['392726', '392739', '392769', '392771', '392774', '392775', '392776', '392798', '392799', '392800', '392802', '392813', '392815', '413062', '413065', '413288', '413289', '414102']
nosy_count = 4.0
nosy_names = ['asvetlov', 'yselivanov', 'pablogsal', 'kumaraditya']
pr_nums = ['17975', '25840', '25842', '25846', '25848', '31275', '31597']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue44011'
versions = ['Python 3.11']

[asvetlov]

There is a PR created a long time ago.
Finally, I've ported tests for it also.

The documentation doesn't mention new ssh_shutdown_timeout parameter yet.

The latest changes from MagicStack/uvloop#385 can be applied separately.

[asvetlov]

New changeset 5fb06ed by Andrew Svetlov in branch 'master':
bpo-44011: New asyncio ssl implementation (bpo-17975)
5fb06ed

[tiran]

The commit has broken multiple build bots, e.g .https://buildbot.python.org/all/#/builders/345/builds/134/steps/5/logs/stdio The new code is missing checks for presence of ssl module. It's an optional component.

[tirkarthi]

The PR made sslproto a hard dependency that even import asyncio fails on non-ssl builds and thus anything that indirectly import asyncio also fails. It seems some of the test modules can be skipped. Some parts of the asyncio codebase already has checks for ssl and has to be done for new parts. Attached is a patch to add more checks but it will be helpful to ensure only relevant parts that absolutely require ssl are skipped.

The test_make_socket_transport is slightly tricky since it tries to simulate ssl being not present by patching it but mock does import of sslproto which will fail since SSLAgainErrors is initialized at module level. Perhaps the test can be modified better to only mock if ssl is not present.

diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index e54ee309e4..6ccac76dfb 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -41,13 +41,14 @@
 from . import exceptions
 from . import futures
 from . import protocols
-from . import sslproto
 from . import staggered
 from . import tasks
 from . import transports
 from . import trsock
 from .log import logger
 
+if ssl is not None:
+ from . import sslproto
 
 __all__ = 'BaseEventLoop',
 
diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py
index 10852afe2b..ac0dc1978c 100644
--- a/Lib/asyncio/proactor_events.py
+++ b/Lib/asyncio/proactor_events.py
@@ -19,11 +19,17 @@
 from . import futures
 from . import exceptions
 from . import protocols
-from . import sslproto
 from . import transports
 from . import trsock
 from .log import logger
 
+try:
+ import ssl
+except ImportError: # pragma: no cover
+ ssl = None
+
+if ssl is not None:
+ from . import sslproto
 
 def _set_socket_extra(transport, sock):
 transport._extra['socket'] = trsock.TransportSocket(sock)
@@ -826,6 +832,9 @@ def loop(f=None):
 server, addr, conn)
 protocol = protocol_factory()
 if sslcontext is not None:
+ if ssl is None:
+ raise RuntimeError('Python ssl module is not available')
+
 self._make_ssl_transport(
 conn, protocol, sslcontext, server_side=True,
 extra={'peername': addr}, server=server,
diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py
index 63ab15f30f..9bc9a03699 100644
--- a/Lib/asyncio/selector_events.py
+++ b/Lib/asyncio/selector_events.py
@@ -23,11 +23,12 @@
 from . import events
 from . import futures
 from . import protocols
-from . import sslproto
 from . import transports
 from . import trsock
 from .log import logger
 
+if ssl is not None:
+ from . import sslproto
 
 def _test_selector_event(selector, fd, event):
 # Test if the selector is monitoring 'event' events
@@ -213,6 +214,9 @@ def _accept_connection(
 protocol = protocol_factory()
 waiter = self.create_future()
 if sslcontext:
+ if ssl is None:
+ raise RuntimeError('Python ssl module is not available')
+
 transport = self._make_ssl_transport(
 conn, protocol, sslcontext, waiter=waiter,
 server_side=True, extra=extra, server=server,
diff --git a/Lib/test/test_asyncio/test_selector_events.py b/Lib/test/test_asyncio/test_selector_events.py
index 349e4f2dca..6aaa7a86be 100644
--- a/Lib/test/test_asyncio/test_selector_events.py
+++ b/Lib/test/test_asyncio/test_selector_events.py
@@ -70,6 +70,7 @@ def test_make_socket_transport(self):
 
 close_transport(transport)
 
+ @unittest.skipIf(ssl is None, 'No ssl module')
 @mock.patch('asyncio.selector_events.ssl', None)
 @mock.patch('asyncio.sslproto.ssl', None)
 def test_make_ssl_transport_without_ssl_error(self):
diff --git a/Lib/test/test_asyncio/test_ssl.py b/Lib/test/test_asyncio/test_ssl.py
index 38235c63e0..c58346bcab 100644
--- a/Lib/test/test_asyncio/test_ssl.py
+++ b/Lib/test/test_asyncio/test_ssl.py
@@ -1,3 +1,8 @@
+from test.support import import_helper
+
+# Skip tests if we don't have ssl
+import_helper.import_module('ssl')
+
 import asyncio
 import asyncio.sslproto
 import contextlib
diff --git a/Lib/test/test_asyncio/test_sslproto.py b/Lib/test/test_asyncio/test_sslproto.py
index 79a81bd8c3..2edbb11b58 100644
--- a/Lib/test/test_asyncio/test_sslproto.py
+++ b/Lib/test/test_asyncio/test_sslproto.py
@@ -11,6 +11,9 @@
 except ImportError:
 ssl = None
 
+# Skip tests if we don't have ssl
+support.import_helper.import_module('ssl')
+
 import asyncio
 from asyncio import log
 from asyncio import protocols

[tiran]

PR #70027 fixes most issues. Gentoo with X buildbot https://buildbot.python.org/all/#builders/465/builds/23 has one failing test.

======================================================================
ERROR: test_create_server_ssl_over_ssl (test.test_asyncio.test_ssl.TestSSL)
----------------------------------------------------------------------

asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.installed/build/target/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
 fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
 File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.installed/build/target/lib/python3.10/test/test_asyncio/test_ssl.py", line 1157, in test_create_server_ssl_over_ssl
 self.loop.run_until_complete(start_server())
 File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.installed/build/target/lib/python3.10/asyncio/base_events.py", line 644, in run_until_complete
 return future.result()
 File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.installed/build/target/lib/python3.10/test/test_asyncio/test_ssl.py", line 1150, in start_server
 await asyncio.wait_for(asyncio.gather(*tasks), TIMEOUT)
 File "/buildbot/buildarea/cpython/pull_request.ware-gentoo-x86.installed/build/target/lib/python3.10/asyncio/tasks.py", line 460, in wait_for
 raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

---

[tiran]

New changeset 37ebdf0 by Christian Heimes in branch 'master':
bpo-44011: Fix asyncio tests without ssl module (GH-25840)
37ebdf0

[tiran]

I have merged my PR to unblock buildbots. Karthikeyan has made suggestions how to improve the tests even further. CI also had some issues with OpenSSL 3.0.0-alpha15. Please run the tests with new OpenSSL version, too. "make multissltest" automates download, compilation, local installation, and testing.