httplib/http.client in method _tunnel used HTTP/1.0 CONNECT method

[vo-va]

BPO	22708
Nosy	@orsenthil, @bitdancer, @berkerpeksag, @vadmium, @demianbrecht, @vo-va
PRs	bpo-22708: Fix http protocol version in CONNECT method, support IDN #742 gh-66897: Upgrade HTTP CONNECT to protocol HTTP/1.1 #8305
Files	py2.7.httplib.patch: pach for python 2.7 issue22708.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/orsenthil'
closed_at = None
created_at = <Date 2014-10-23.06:21:23.450>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'httplib/http.client in method _tunnel used HTTP/1.0 CONNECT method'
updated_at = <Date 2021-11-07.18:37:03.856>
user = 'https://github.com/vo-va'

bugs.python.org fields:

activity = <Date 2021-11-07.18:37:03.856>
actor = 'eric.araujo'
assignee = 'orsenthil'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2014-10-23.06:21:23.450>
creator = 'vova'
dependencies = []
files = ['36996', '38808']
hgrepos = []
issue_num = 22708
keywords = ['patch']
message_count = 6.0
messages = ['229856', '229876', '229915', '239921', '239952', '241237']
nosy_count = 6.0
nosy_names = ['orsenthil', 'r.david.murray', 'berker.peksag', 'martin.panter', 'demian.brecht', 'vova']
pr_nums = ['742', '8305']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22708'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[vo-va]

At my workplace I have to use corporate Internet proxy server with AD/domain/ntlm authorization. I use local cntlm proxy server to authorize myself on corporate proxy. Programs are send requests to cntlm proxy without any authorization information. Cntlm proxy communicate with corporate proxy and handle all authorization stuff and return response to programs.

But programs which use httplib, like pip, and want to open https url can't work in my network scheme. Because to open https connection httplib send to cntlm proxy

"CONNECT encrypted.google.com:443 HTTP/1.0\r\n"

HTTP/1.0 does not assume persistent connection so corporate proxy return http response 407 (need authorization) and close connection. Cntlm proxy detect closed connection and return http response 407 to pip/httplib which can't handle this response or begin ntlm negotiation, throw exception

ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',)) 

and close.

So I suggest change HTTP CONNECT method to

"CONNECT %s:%d HTTP/1.1\r\n"

This change allow cntlm proxy keep alive connection to corporate proxy do all authorization stuff and return proper response.

And also in header of httplib is stated what it is "HTTP/1.1 client library"

[bitdancer]

See bpo-21224 and bpo-9740. It's not 100% clear to me from reading those issues, but it *sounds* like the support is there, you just have to turn it on. So I'm closing this issue as not a bug...either it already works (and you just have to set your code to use 1.1) or this is a duplicate of one of those two issues, and you should comment on whichever one is more appropriate.

Your point about PIP is a good one, though...that might be worth a specific issue for 1.1 support in pip, if it doesn't already have a way to switch it on.

[vo-va]

The issue http://bugs.python.org/issue21224 is about http server implementations. The issue http://bugs.python.org/issue9740 is more relevant for what I talking about, but not exactly.

Look, in this line

https://hg.python.org/cpython/file/6a2f74811240/Lib/http/client.py#l786

http protocol version is setted and in this line

https://hg.python.org/cpython/file/6a2f74811240/Lib/http/client.py#l1036

variable used to send http method (GET, PUT etc) and it work for direct connection or proxy with http connections. But if required to use CONNECT method through proxy (usually used for https connection) will be used _tunnel() method from http.client (py3k) or httplib (py2.7)

https://hg.python.org/cpython/file/6a2f74811240/Lib/http/client.py#l871

and here version off http is hardcoded to HTTP/1.0

PIP use urllib3, but urllib3 for actual network working (work with socket and send request) use httplib or http.client. So I think it would be better to make changes in httplib than override _tunnel() method in urllib3.

P.S. I'm not sure about rules how to open/close issues, so I open this issue again. I'm sorry if this causes some inconvenience.

[demianbrecht]

Thanks for the report and follow up Vova. I've come across this line and it puzzled me as well as to why it's hardcoded. Rather than the hardcoded approach in your patch, I've attached a patch that uses _http_vsn_str which is consistent with other areas of the library.

Additionally, I noticed that a failure will occur if the destination address contains characters outside of ASCII range, so I've added support for IDN.

I assume that the patch should be committed to tip and maintenance branches as both issues are bugs (there is IDN support elsewhere in the library).

[vadmium]

Patch looks good to me. I’m not an expert on non-ASCII domain names, but enabling HTTP 1.1 should be okay, at least for new versions of Python. CONNECT for HTTP 1.1 is described at <https://tools.ietf.org/html/rfc7231#section-4.3.6\>.

Also, see the code review about removing an comment about HTTP 1.1 blessing.

[vo-va]

I don't know why I hardcoded version of HTTP, it was almost 6 months ago. Maybe because it just POC, or maybe because HTTP 1.0 is not allowed CONNECT method. Thank you for your patch and also for IDN support.