ctypes.util.find_library raises FileNotFoundError for libsecp256k1

[relativisticelectron]

Bug report

The following script

import ctypes
from ctypes import util

print(f'ctypes version{ctypes.__version__}')


print('Test foofoofoo')
print(f"returns:{util.find_library('foofoofoo')}\n")

print('Test secp256k1')
print(f"returns:{util.find_library('secp256k1')}\n")

print('Test libsecp256k1')
print(f"returns:{util.find_library('libsecp256k1')}\n")

gives the output:

ctypes version 1.1.0
Test foofoofoo
returns: None

Test secp256k1
returns: libsecp256k1.so.0

Test libsecp256k1
Traceback (most recent call last):
 File "/home/<user>/programs/testctypes.py", line 14, in <module>
 print(f"returns:{util.find_library('libsecp256k1')}\n")
 File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/ctypes/util.py", line 351, in find_library
 _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
 File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/ctypes/util.py", line 148, in _findLib_gcc
 if not _is_elf(file):
 File "/home/<user>/programs/anaconda3/envs/python310/lib/python3.10/ctypes/util.py", line 100, in _is_elf
 with open(filename, 'br') as thefile:
FileNotFoundError: [Errno 2] No such file or directory: b'liblibsecp256k1.a'

find_library should not fail, but return None if nothing was found. In Ubuntu 20.04 this script runs without any exception.

Your environment

• ctypes version 1.1.0
• Operating system and architecture: Kubuntu 22.04 LTS x86_64, linux 5.15.0-27-generic
• Bug can be reproduced in python versions: python 3.10 (via anaconda and via ubuntu deb repositories), python 3.8 (via anaconda)
• The installed libsecp256k1 packages are
  

contains the files

/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libsecp256k1.so.0
/usr/lib/x86_64-linux-gnu/libsecp256k1.so.0.0.0
/usr/share
/usr/share/doc
/usr/share/doc/libsecp256k1-0
/usr/share/doc/libsecp256k1-0/changelog.Debian.gz
/usr/share/doc/libsecp256k1-0/copyright

And contains the files

/.
/usr
/usr/include
/usr/include/secp256k1.h
/usr/include/secp256k1_ecdh.h
/usr/include/secp256k1_extrakeys.h
/usr/include/secp256k1_preallocated.h
/usr/include/secp256k1_recovery.h
/usr/include/secp256k1_schnorrsig.h
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libsecp256k1.a
/usr/lib/x86_64-linux-gnu/libsecp256k1.so
/usr/lib/x86_64-linux-gnu/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig/libsecp256k1.pc
/usr/share
/usr/share/doc
/usr/share/doc/libsecp256k1-dev
/usr/share/doc/libsecp256k1-dev/changelog.Debian.gz
/usr/share/doc/libsecp256k1-dev/copyright

And in /usr/lib/x86_64-linux-gnu/ are the following relevant files containing secp256k1:

This bug was noticed in cryptoadvance/specter-desktop#1710

[relativisticelectron]

When I do a print(trace) just before

cpython/Lib/ctypes/util.py

Line 139 in 8241a69

res = re.findall(expr, trace)

I get

b'/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o\n/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crti.o\n/usr/lib/gcc/x86_64-linux-gnu/11/crtbeginS.o\n/usr/bin/ld: cannot find -llibsecp256k1: No such file or directory\n/usr/bin/ld: note to link with /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/libsecp256k1.a use -l:libsecp256k1.a or rename it to liblibsecp256k1.a\n/usr/lib/gcc/x86_64-linux-gnu/11/libgcc.a\n/usr/lib/gcc/x86_64-linux-gnu/11/libgcc_s.so\n/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/libgcc_s.so.1\n/usr/lib/gcc/x86_64-linux-gnu/11/libgcc.a\ncollect2: error: ld returned 1 exit status\n'

here with replaced \n

b'/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o
/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/11/crtbeginS.o
/usr/bin/ld: cannot find -llibsecp256k1: No such file or directory
/usr/bin/ld: note to link with /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/libsecp256k1.a use -l:libsecp256k1.a or rename it to liblibsecp256k1.a
/usr/lib/gcc/x86_64-linux-gnu/11/libgcc.a
/usr/lib/gcc/x86_64-linux-gnu/11/libgcc_s.so
/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/libgcc_s.so.1
/usr/lib/gcc/x86_64-linux-gnu/11/libgcc.a
collect2: error: ld returned 1 exit status
'

My interpretation of this is, that the re logic in

cpython/Lib/ctypes/util.py

Line 139 in 8241a69

res = re.findall(expr, trace)

is not sufficient to handle the ld message.

[cz6ace]

I face the same issue, when the output is polluted with message from ld, in my case:
/usr/bin/ld: note to link with /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/libusb-1.0.a use -l:libusb-1.0.a or rename it
reproduced with using pyInstaller package, but can be reproduced easily, see below.

Environment:
Ubuntu 22.04 LTS
uname: Linux ubu22-virtual 5.15.0-50-generic #56-Ubuntu SMP Tue Sep 20 13:23:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Python: 3.10.6

Steps to reproduce:

sudo apt install libusb-1.0-0-dev
/usr/bin/gcc -Wl,-t -o /tmp/tmpp455oz01 -llibusb-1.0

outputting

/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/Scrt1.o
/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/crti.o
/usr/lib/gcc/x86_64-linux-gnu/11/crtbeginS.o
/usr/bin/ld: cannot find -llibusb-1.0: No such file or directory
/usr/bin/ld: note to link with /usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/libusb-1.0.a use -l:libusb-1.0.a or rename it to liblibusb-1.0.a
/usr/lib/gcc/x86_64-linux-gnu/11/libgcc.a
/usr/lib/gcc/x86_64-linux-gnu/11/libgcc_s.so
/usr/lib/gcc/x86_64-linux-gnu/11/../../../x86_64-linux-gnu/libgcc_s.so.1
/usr/lib/gcc/x86_64-linux-gnu/11/libgcc.a
collect2: error: ld returned 1 exit status

in python3

from ctypes import util
util.find_library('libusb-1.0')

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/usr/lib/python3.10/ctypes/util.py", line 341, in find_library
 _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
 File "/usr/lib/python3.10/ctypes/util.py", line 147, in _findLib_gcc
 if not _is_elf(file):
 File "/usr/lib/python3.10/ctypes/util.py", line 99, in _is_elf
 with open(filename, 'br') as thefile:
FileNotFoundError: [Errno 2] No such file or directory: b'liblibusb-1.0.a'

If the gcc produces only the list of files, then prepending ^ as looking for whole line could fix the issue.
expr = os.fsencode(r'[^\(\)\s]*lib%s\.[^\(\)\s]*' % re.escape(name))
to
expr = os.fsencode(r'^[^\(\)\s]*lib%s\.[^\(\)\s]*' % re.escape(name))