urllib does not send http/1.1 ALPN extension

[pmenzel]

BPO	40968
Nosy	@orsenthil, @tiran
PRs	bpo-40968: Send http/1.1 ALPN extension #20959

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-06-13.11:10:14.395>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.7', 'library']
title = 'urllib does not send http/1.1 ALPN extension'
updated_at = <Date 2020-11-13.15:38:00.535>
user = 'https://bugs.python.org/pmenzel'

bugs.python.org fields:

activity = <Date 2020-11-13.15:38:00.535>
actor = 'christian.heimes'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-06-13.11:10:14.395>
creator = 'pmenzel'
dependencies = []
files = []
hgrepos = []
issue_num = 40968
keywords = ['patch']
message_count = 5.0
messages = ['371450', '371619', '371620', '371621', '380899']
nosy_count = 3.0
nosy_names = ['orsenthil', 'christian.heimes', 'pmenzel']
pr_nums = ['20959']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40968'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

[pmenzel]

Having the TURN server Coturn 1 set up in a Jitsi Meet installation, Python’s urllib requests fail, while it works with cURL and browsers.

$ curl -I https://jitsi.molgen.mpg.de
HTTP/2 200 
server: nginx/1.14.2
date: Sat, 13 Jun 2020 11:09:19 GMT
content-type: text/html
vary: Accept-Encoding
strict-transport-security: max-age=63072000

>>> import urllib.request
>>> response = urllib.request.urlopen('https://jitsi.molgen.mpg.de')
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
 return opener.open(url, data, timeout)
 File "/usr/lib/python3.8/urllib/request.py", line 525, in open
 response = self._open(req, data)
 File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
 result = self._call_chain(self.handle_open, protocol, protocol +
 File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
 result = func(*args)
 File "/usr/lib/python3.8/urllib/request.py", line 1393, in https_open
 return self.do_open(http.client.HTTPSConnection, req,
 File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
 r = h.getresponse()
 File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
 response.begin()
 File "/usr/lib/python3.8/http/client.py", line 303, in begin
 version, status, reason = self._read_status()
 File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
 raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

[tiran]

It looks like the server is refusing requests that don't have an ALPN extension set in TLS layer. At first I though that the server may only support HTTP/2. The curl requests in your examples uses HTTP/2. urllib only supports HTTP/1.1 and HTTP/1.0.

Then I tried "curl --http1.1 https://jitsi.molgen.mpg.de". The request also succeeds because curl is sending "ALPN, offering http/1.1".

This request works for me:

>>> import ssl
>>> import urllib.request
>>> ctx = ssl.create_default_context()
>>> ctx.set_alpn_protocols(['http/1.1'])
>>> urllib.request.urlopen('https://jitsi.molgen.mpg.de', context=ctx)
<http.client.HTTPResponse object at 0x6040009b7960>

urllib could set the ALPN header by default when the user does not supply a custom context. It looks like curl always adds an ALPN extension. I don't see code in Python requests that sets ALPN extension. On the other hand Tom Christie's excellent httpx library does set ALPN extension just like curl.

[pmenzel]

Wow, great job in figuring this out. Passing the context, it works here too.

I just wanted to add, that I looked at this with Wireshark, and to not complicate things, first tried plain HTTP.

$ curl http://jitsi.molgen.mpg.de
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.14.2</center>
</body>
</html>

With Python, there is also the connection lost error. But Wireshark shows that the 301 HTTP packet is send back by the server.

$ python3
Python 3.8.3 (default, May 14 2020, 11:03:12) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> response = urllib.request.urlopen('http://jitsi.molgen.mpg.de')
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
 return opener.open(url, data, timeout)
 File "/usr/lib/python3.8/urllib/request.py", line 531, in open
 response = meth(req, response)
 File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
 response = self.parent.error(
 File "/usr/lib/python3.8/urllib/request.py", line 563, in error
 result = self._call_chain(*args)
 File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
 result = func(*args)
 File "/usr/lib/python3.8/urllib/request.py", line 755, in http_error_302
 return self.parent.open(new, timeout=req.timeout)
 File "/usr/lib/python3.8/urllib/request.py", line 525, in open
 response = self._open(req, data)
 File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
 result = self._call_chain(self.handle_open, protocol, protocol +
 File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
 result = func(*args)
 File "/usr/lib/python3.8/urllib/request.py", line 1393, in https_open
 return self.do_open(http.client.HTTPSConnection, req,
 File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
 r = h.getresponse()
 File "/usr/lib/python3.8/http/client.py", line 1332, in getresponse
 response.begin()
 File "/usr/lib/python3.8/http/client.py", line 303, in begin
 version, status, reason = self._read_status()
 File "/usr/lib/python3.8/http/client.py", line 272, in _read_status
 raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

[tiran]

Look closer at the traceback. You'll see that urllib follows redirects (http_response -> http_error_302 -> open -> HTTPSConnection). curl doesn't follow redirects by default.