ImportError: _ssl.so: undefined symbol: _PyModule_Add

[0x2b3bfa0]

Bug report

Environment

$ uname
Linux
$ python --version{,}
Python 3.11.4 (tags/v3.11.4:d2340ef257, Aug 26 2023, 22:42:28) [GCC 12.2.0]

Description

Importing the ssl shared library from a Python 3.11.5 build from a Python 3.11.4 binary produces an ImportError due to an undefined symbol. Isn't CPython's ABI supposed to be fully compatible between patch releases?

CPython’s Application Binary Interface (ABI) is forward- and backwards-compatible across a minor release.

Example

FROM debian

RUN apt update\
 && apt install --yes build-essential git libssl-dev zlib1g-dev

RUN git clone --branch v3.11.4 https://github.com/python/cpython 3.11.4\
 && cd 3.11.4 && ./configure && make

RUN git clone --branch v3.11.5 https://github.com/python/cpython 3.11.5\
 && cd 3.11.5 && ./configure && make

RUN mv 3.11.5/build/lib.linux-x86_64-3.11/_ssl.cpython-311-x86_64-linux-gnu.so\
 3.11.4/build/lib.linux-x86_64-3.11/_ssl.cpython-311-x86_64-linux-gnu.so

RUN 3.11.4/python -m ssl

Error

Traceback (most recent call last):
 File "<frozen runpy>", line 198, in _run_module_as_main
 File "<frozen runpy>", line 88, in _run_code
 File "//3.11.2/Lib/ssl.py", line 100, in <module>
 import _ssl # if we can't import it, let the error propagate
 ^^^^^^^^^^^
ImportError: //3.11.2/build/lib.linux-x86_64-3.11/_ssl.cpython-311-x86_64-linux-gnu.so: undefined symbol: _PyModule_Add

Observations

It looks like fced79f is the breaking commit, as per git bisect results.

Linked PRs

• gh-108525: Clarify stable ABI documentation to exclude internal C libraries #108582

[hansegucker]

I also observed the issue on my system.

[CorvinM]

Also able to reproduce between 3.11 branch and 3.11.4, looks like #106863 broke the ABI

[sunmy2019]

@serhiy-storchaka as the commit author

[serhiy-storchaka]

I don't think this warranty extends to the private API.

Names prefixed by an underscore, such as _Py_InternalState, are private API that can change without notice even in patch releases.

If you only use the public C API in your extension, you can expect forward- and backwards-compatibinility.

[Templarrr]

I see the same error locally in poetry after my Ubuntu upgraded 3.11.? -> 3.11.5. Any advice how this can be mitigated/fixed? It doesn't look like I can downgrade local python back to 3.11.4 - deadsnakes ppa don't list 3.11.4 anymore

[LostInDarkMath]

@Templarrr I had the same issue and fixed it by updating poetry to the latest version.

[erlend-aasland]

cc. RM @pablogsal

[Templarrr]

@LostInDarkMath thank you! Seems to be the solution

[encukou]

The reproducer looks contrived. In what situation would pieces of CPython get mixed up like this? Is Ubuntu's _ssl module not built together with the interpreter?

[CorvinM]

The reproducer looks contrived. In what situation would pieces of CPython get mixed up like this? Is Ubuntu's _ssl module not built together with the interpreter?

A faulty install or packaging script could easily cause this (but wouldn't be noticed in most cases, since they are usually ABI compatible) because the file paths are named identically (I.e., specific to 3.11, not 3.11.5).