OpenSSL library collision with external python on linux systems

[hhaensel]

Affects: PythonCall

Describe the bug
When both OpenSSL and PythonCall are loaded pyimport("ssl") fails if an external python executable is used.

julia> ENV["JULIA_PYTHONCALL_EXE"] = joinpath(ENV["MAMBA_ROOT_PREFIX"], "bin", "python")
julia> using OpenSSL, PythonCall
julia> println(pyimport("sys").executable)
julia> pyimport("ssl")

/opt/conda/bin/python
ERROR: Python: ImportError: /home/mambauser/.julia/artifacts/625d0c7f43b3ae48c9d49a1a17970e8e9d7e271b/lib/libcrypto.so: version `OPENSSL_3.3.0' not found (required by /opt/conda/lib/python3.10/lib-dynload/_ssl.cpython-310-aarch64-linux-gnu.so)
Python stacktrace:
 [1] <module>
 @ /opt/conda/lib/python3.10/ssl.py:99

Obviously, the python import tries to load Julia's OpenSSL cryptolibrary although the calling python executable is correctly set.
A workaround is to import the ssl module before loading OpenSSL. However, I am not sure whether Julia's OpenSSL will function properly.

julia> ENV["JULIA_PYTHONCALL_EXE"] = joinpath(ENV["MAMBA_ROOT_PREFIX"], "bin", "python")
julia> using PythonCall
julia> println(pyimport("sys").executable)
julia> using OpenSSL
julia> pyimport("ssl")

Python: <module 'ssl' from '/opt/conda/lib/python3.10/ssl.py'>

Interestingly, I didn't succeed in using the CondaPkg's python version:

julia> ENV["JULIA_CONDAPKG_BACKEND"] = "MicroMamba";
julia> using PythonCall
(@v1.10) pkg> conda st
CondaPkg Status /home/mambauser/.julia/environments/v1.10/CondaPkg.toml
Environment
 /home/mambauser/.julia/environments/v1.10/.CondaPkg/env
Packages
 pandas v2.2.2

julia> pyimport("sys").executable
Python: '/opt/conda/bin/python'
julia> pyimport("ssl")
Python: <module 'ssl' from '/opt/conda/lib/python3.10/ssl.py'>

My system

• Docker: micromamba 1.5.10 (Ubuntu)
• Julia 1.10.5, Python 3.10, PythonCall 0.9.23, OpenSSL 1.4.3

julia> versioninfo()
Julia Version 1.10.5
Commit 6f3fdf7b362 (2024-08-27 14:19 UTC)
Build Info:
 Official https://julialang.org/ release
Platform Info:
 OS: Linux (aarch64-linux-gnu)
 CPU: 2 × unknown
 WORD_SIZE: 64
 LIBM: libopenlibm
 LLVM: libLLVM-15.0.7 (ORCJIT, generic)
Threads: 1 default, 0 interactive, 1 GC (on 2 virtual cores)
Environment:
 JULIA_REVISE = on
 JULIA_FILE = julia-1.10.5-linux-x86_64.tar.gz
 JULIA_CONDAPKG_BACKEND = MicroMamba
 JULIA_PYTHONCALL_EXE = /opt/conda/bin/python
 JULIA_CC = /opt/conda/bin/g++
 JULIA_DEPOT_PATH = /home/mambauser/.julia

(@v1.10) pkg> st
Status `~/.julia/environments/v1.10/Project.toml`
 [4d8831e6] OpenSSL v1.4.3
 [6099a3de] PythonCall v0.9.23


(@v1.10) pkg> conda st
CondaPkg Status /home/mambauser/.julia/environments/v1.10/CondaPkg.toml
Using the Null backend (dependencies shown here are not being managed)
Environment
 
Packages
 pandas

[hhaensel]

The better workaround might be to install the identical versions of openssl, if you have the freedom to do so.
In my case the following installation made the error disappear:

micromamba install -c conda-forge openssl=3.3.0

as the error above contained version OPENSSL_3.3.0' not found`.
This is still a bit puzzling to me, as the current latest OpenSSL_jll version is 3.0.15, but at least I could make my program run again.
Hope this can be of help for people with similar library collisions.

[cjdoris]

Yep as you've found, if you use Julia and Python packages that both rely on the same underlying library, it's often important to ensure the packages are at somewhat compatible versions so that the library they actually use is not to different than the one they expect.

If you use PythonCall with the default settings then CondaPkg will try to install a version of Python OpenSLL that is compatible with Julia's OpenSSL. If you use your own Python environment then it's up to you to ensure this compatibility.

It's possible we could insert a warning somewhere for common cases of incompatible libraries.

[hhaensel]

Thanks for your comment and good to know that care has been taken to provide best possible compatibility. I might change my Docker containers to rely on CondaPkg rather than on micromamba then.