Failures in SNI support affecting pip

[tysonite]

We are hitting this error when trying to install any package using pip shipped with Jython:

~/robotframework$ jython/bin/jython -m pip -vvv install docutils
DEPRECATION: A future version of pip will drop support for Python 2.7.
Created temporary directory: /tmp/pip-ephem-wheel-cache-zG0Xvb
Created temporary directory: /tmp/pip-req-tracker-8ZbSlA
Created requirements tracker '/tmp/pip-req-tracker-8ZbSlA'
Created temporary directory: /tmp/pip-install-LirEp0
Collecting docutils
 1 location(s) to search for versions of docutils:
 * https://pypi.org/simple/docutils/
 Getting page https://pypi.org/simple/docutils/
 Looking up "https://pypi.org/simple/docutils/" in the cache
 Request header has "max_age" as 0, cache bypassed
 Starting new HTTPS connection (1): pypi.org:443
 https://pypi.org:443 "GET /simple/docutils/ HTTP/1.1" 403 296
 Status code 403 not in (200, 203, 300, 301)
 Could not fetch URL https://pypi.org/simple/docutils/: 403 Client Error: [[[!!! BREAKING CHANGE !!!]]] Support for clients that do not support Server Name Indication is temporarily disabled and will be permanently deprecated soon. See https://status.python.org/incidents/hzmjhqsdjqgb and https://github.com/pypa/pypi-support/issues/978 [[[!!! END BREAKING CHANGE !!!]]] for url: https://pypi.org/simple/docutils/ - skipping
 ERROR: Could not find a version that satisfies the requirement docutils (from versions: none)
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-8ZbSlA'
ERROR: No matching distribution found for docutils
Exception information:
Traceback (most recent call last):
 File "/home/tysonite/robotframework/jython/Lib/site-packages/pip/_internal/cli/base_command.py", line 178, in main
 status = self.run(options, args)
 File "/home/tysonite/robotframework/jython/Lib/site-packages/pip/_internal/commands/install.py", line 352, in run
 resolver.resolve(requirement_set)
 File "/home/tysonite/robotframework/jython/Lib/site-packages/pip/_internal/commands/install.py", line 352, in run
 resolver.resolve(requirement_set)
 File "/home/tysonite/robotframework/jython/Lib/site-packages/pip/_internal/resolve.py", line 130, in resolve
 discovered_reqs.extend(
 File "/home/tysonite/robotframework/jython/Lib/site-packages/pip/_internal/resolve.py", line 130, in resolve
 discovered_reqs.extend(
 File "/home/tysonite/robotframework/jython/Lib/site-packages/pip/_internal/resolve.py", line 294, in _resolve_one
 abstract_dist = self._get_abstract_dist_for(req_to_install)
 File "/home/tysonite/robotframework/jython/Lib/site-packages/pip/_internal/resolve.py", line 240, in _get_abstract_dist_for
 abstract_dist = self.preparer.prepare_linked_requirement(
 File "/home/tysonite/robotframework/jython/Lib/site-packages/pip/_internal/operations/prepare.py", line 288, in prepare_linked_requirement
 req.populate_link(finder, upgrade_allowed, require_hashes)
 File "/home/tysonite/robotframework/jython/Lib/site-packages/pip/_internal/req/req_install.py", line 198, in populate_link
 self.link = finder.find_requirement(self, upgrade)
 File "/home/tysonite/robotframework/jython/Lib/site-packages/pip/_internal/index.py", line 790, in find_requirement
 raise DistributionNotFound(
DistributionNotFound: No matching distribution found for docutils
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 403 296
Status code 403 not in (200, 203, 300, 301)
Could not fetch URL https://pypi.org/simple/pip/: 403 Client Error: [[[!!! BREAKING CHANGE !!!]]] Support for clients that do not support Server Name Indication is temporarily disabled and will be permanently deprecated soon. See https://status.python.org/incidents/hzmjhqsdjqgb and https://github.com/pypa/pypi-support/issues/978 [[[!!! END BREAKING CHANGE !!!]]] for url: https://pypi.org/simple/pip/ - skipping

Is there any way to bypass that?

Java is:

$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~18.04-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)

Jython is:

$ jython/bin/jython -V
Jython 2.7.2

[underscoreglook]

I am also running into this issue.

[jbrzozoski]

Hit the same lack-of-SNI error on my system today.

$ java -version
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)
OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.20.04, mixed mode, sharing)
$ ~/jython2.7.2/bin/jython -V
Jython 2.7.2

[jeff5]

A couple of useful references.

From the error message, the short-notice change to PyPI: pypi/support#978 . (I have not read all this, but notice what may be a work-around in: pypi/support#978 (comment) .

Wikipedia background: https://en.wikipedia.org/wiki/Server_Name_Indication .

I think the solution lies in porting SNI function from a later CPython library. Unfortunately this area is heavily customised for Jython and ssl.py at least looks like 90% re-write of its CPython counterpart. No quick fix there.

[jbrzozoski]

In my case, a simpler workaround was to clone the source for the library I was trying to install, and then from the source directory run:

$ jython setup.py install

It automatically fetched and installed all the relevant dependency libraries as well.

It didn't fix the SNI issue, but it got the libraries installed, which was all I really needed.

[jeff5]

A few notes. The simplest way to reproduce is:

PS 273a1-trial> inst\bin\jython -m pip -v --no-color list
DEPRECATION: A future version of pip will drop support for Python 2.7.
Package Version Location Installer
---------- ------- ----------------------------------------------------------------- ---------
pip 19.1 c:\users\jeff\documents\jython\273a1-trial\inst\lib\site-packages pip
setuptools 41.0.1 c:\users\jeff\documents\jython\273a1-trial\inst\lib\site-packages pip
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
ERROR: Certificate did not match expected hostname: pypi.org. Certificate:{'notAfter': 'Nov 23 18:41:10 2022 GMT', 'subject': ((('commonName', 'www.python.org'),),), 'subjectAltName': (('DNS', 'www.python.org'), ('DNS', '*.python.org'), ('DNS', 'docs.python.org'), ('DNS', 'downloads.python.org'), ('DNS', 'pypi.python.org'))}

However, you get an actual stack dump if you try to install which may contain more clues.

It's not specific to Java 8. (I got identical results in 11 and 17).

CPython 2.7.16 works correctly with pypi.org and pip 19.1 the same version as Jython. An odd difference is that no amount of -v seems able to make the logging messages I see from pip on Jython emerge from pip on CPython.

I spent some time exploring the code of pip. One can find where the message Starting new HTTPS connection is produced. Static analysis didn't get me very far because of the amount of pluggable behaviour where one cannot tell statically what will be plugged in. The code that fails will be a long way from the code that reports the message. I need to insert more debug or logging into a personal copy.

I currently think _socket.py and ssl.py are the places to look for our shortcomings wrt SNI.

pip provides a lot of logic that appears to duplicate features in socket. I wonder if the difference in what gets logged is due to Jython not providing those features and pip falling back to its own implementation.

Our socket and ssl implementations show definite signs of incompleteness (counting FIXME and commented out sections). I have never more than scratched the surface of the socket module to address some encoding problems. We have customised tests and disable the CPython ones, so it is unlikely to be conformant.

[jeff5]

Just a few working notes.

One can exercise this in the development environment after installing pip:

dist\bin\jython -m ensurepip

In executing the command:

dist\bin\jython -m pip -v --no-color list

or the same with python, the list only needs local information to run and no networking. The fault is exhibited at the point it checks pypi.org for a later version of pip. A working pip will only do so once a week, writing what it found in the pip cache (varies, see docs), so has to be forced here to do it every time:
https://github.com/pypa/pip/blob/307410c24b82be7bc2e59ff642430bd1d7b4d718/src/pip/_internal/utils/outdated.py#L123

Jython always does this check at present, since it has never succeeded. (Not sure where it keeps the answer.)

Working back from where the problem is detected, as a mismatching certificate, and looking only at the behaviour of Jython, I cannot spot where it goes wrong.

It gets a server certificate, but it is the base one from the hosting service (I think), definitely not the pypi.org certificate it wants. Earlier in the exchange, Jython should have included an SNI header that requests the specific server host name. I can see a server_hostname being passed around, but it is not clear where it should have been used for SNI, or why it wasn't.

Jython socket is essentially in _socket.py and there is no built-in. _socket.py has been written to combine CPython socket.py and socketmodule.c using Java integration. Some code is borrowed, but a static comparison just yields a massive diff.

Since I am looking for the place where Jython doesn't do something, without knowing where it should, I've been comparing CPython 2.7.16 and Jython behaviour under pdb.

Jython executes the pip code faithfully and it is a long time before we hit Jython specific code.

I have found a first divergence where socket.getaddrinfo is called, here:
https://github.com/pypa/pip/blob/307410c24b82be7bc2e59ff642430bd1d7b4d718/src/pip/_vendor/urllib3/util/connection.py#L57

The divergence is because these two constants have been swapped:

jython/Lib/_socket.py

Lines 113 to 114 in 50729e6

	SOCK_DGRAM = 1
	SOCK_STREAM = 2

We should have:

SOCK_STREAM = 1
SOCK_DGRAM = 2

[jeff5]

Jython 2 ssl.py should provide SNI support thanks initially to this work. So you'd think we would be fairly close without this being a new feature. It's just a matter of finding a bug. :-(

I'm still finding it very difficult to pick my way amongst the similarities and huge differences between the Jython 2 and CPython socket and ssl modules. I've steered away from this area until now, and it has been quietly disintegrating. If anyone thinks they understand it, please pitch in. @jimbaker @darjus

The differences in API make comparison of pip's behaviour difficult once we reach this interface. So what do the unit tests tell us? Our test_socket.py is customised, and may actually be richer than CPython's. To test ssl , we run test_ssl_jy.py instead of test_ssl.py. But the former runs only 40 tests compared to 96 in the latter (even after disabling the hanging tests). I'm looking for more of test_ssl.py to pass to reduce or at least understand the API differences.

Unfortunately, Python makes ssl visibly a wrapper on OpenSSL. At the moment I'm picking my way through the ssl.OP_* flags, whose values are OpenSSL API but are re-sold by CPython. Jython rightly uses Java technologies to emulate these details. While the actual bit-positions of flags will not matter if they are referenced symbolically, I think this needs fixing:

jython/Lib/ssl.py

Line 95 in 8c3734d

OP_ALL, OP_NO_SSLv2, OP_NO_SSLv3, OP_NO_TLSv1 = range(4)

Other than the clearly wrong, I don't know if the many divergences I find are worth correcting in the interests of fixing the SNI bug. Changes with the intention only of making it more like CPython could cause more bugs than they avoid. However, I think Jython 3 should be more like CPython 3 than this, for example splitting off _ssl, even if it is written in Python.

[jeff5]

I may have found the problem. Here pip asks for the certificate corresponding to the host name (with context.verify_mode = cert_reqs, which is CERT_REQUIRED in practice), but it disables checking the certificate.
https://github.com/pypa/pip/blob/307410c24b82be7bc2e59ff642430bd1d7b4d718/src/pip/_vendor/urllib3/util/ssl_.py#L283-L287

Here we treat disabling the check as a sign we need not ask for the host-specific certificate, and so the session will only carry that of the host computer.

jython/Lib/ssl.py

Lines 1113 to 1119 in 8c3734d

	if self._check_hostname:
	engine = context_builder.build().newEngine(ByteBufAllocator.DEFAULT, hostname, addr[1])
	if HAS_SNI:
	params = engine.getSSLParameters()
	params.setEndpointIdentificationAlgorithm('HTTPS')
	params.setServerNames([SNIHostName(hostname)])
	engine.setSSLParameters(params)

In other contexts, we use if self.verify_mode != CERT_NONE: as the test. Substituting that seems to do the trick.

That was hard.