freeze cannot find Modules/_hacl/libHacl_Hash_SHA2.a

[chris21k]

Bug report

Bug description:

Dear maintainers,

I am using Ubuntu Noble 24.04 development. I know there are some alternatives to freeze, but I have to maintain a larger project as CGI script which is (for me) not so easy to transform and I use it up to 20 years now.

$ python3 --version
Python 3.12.2
$ dpkg -l | grep python3.12
ii python3.12 3.12.2-1 amd64 Interactive high-level object-oriented language (version 3.12)
...
$ cat helloworld.py
#!/usr/bin/python3

print("Hello, World!")
$ /usr/share/doc/python3.12/examples/freeze/freeze.py -o frozen helloworld.py
...
$ cd frozen
$ make
...
x86_64-linux-gnu-gcc -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Xlinker -expo
rt-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions config.o frozen.o M___future__.o M__
_main__.o ...
M_zipp.o /usr/lib/x86_64-linux-gnu/libpython3.12.so -lm -lm -lz -lm -lm -lexpat -lexpat Modules/_hacl/libHacl_Hash_SHA2.a -lz -ldl -lm -o helloworld
/usr/bin/ld: cannot find Modules/_hacl/libHacl_Hash_SHA2.a: No such file or directory
collect2: error: ld returned 1 exit status
make: *** [Makefile:1236: helloworld] Error 1

The linker does not find the library Modules/_hacl/libHacl_Hash_SHA2.a . As far I can see, Ubuntu does not deliver Modules/_hacl/libHacl_Hash_SHA2.a as a binary anywhere in the system or as another deb-package.

The only option I see so far is to compile the python sources myself and copy/link the static Modules folder to the current directory. The following Dockerfile shows that (after removing RUN and changing WORKDIR to cd it should be also runable as shell script). However, this is very unhandy and runs > 1h on my machine.

FROM docker.io/library/ubuntu:noble

# enable package sources
RUN sed -i '/deb-src/s/^# //' /etc/apt/sources.list && apt-get update

# Ubuntu cloud images do not install all files from packages. This drops, e.g.,
# the Python examples; therefore, remove the Ubuntu cloud "excludes" configuration.
RUN rm -f /etc/dpkg/dpkg.cfg.d/excludes

# install required packages
RUN apt-get --yes install build-essential python3.12-dev python3.12-examples

# build python
RUN apt-get --yes build-dep python3.12
RUN mkdir /build /demo /demo/frozen
WORKDIR /build
RUN apt-get --yes source python3.12
WORKDIR /build/python3.12-3.12.2
RUN DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage --no-sign -b -j16

# create python demo program
WORKDIR /demo
RUN echo '#!/usr/bin/python3\nprint("Hello, World!")' >helloworld.py

# run freeze
RUN /usr/share/doc/python3.12/examples/freeze/freeze.py -o frozen helloworld.py

# add required link to static modules
RUN ln -s /build/python3.12-3.12.2/build-static/Modules /demo/frozen/Modules

# build and run frozen demo program
WORKDIR /demo/frozen
RUN make && ./helloworld

Why is the Hacl library statically linked? Is linking to it necessary at all? Would it be an option to have it as .so as I am used e.g. from psycopg? If needed, in any case it should be provided as binary by the distro. Further, the by freeze generated frozen/Makefile (see Makefile.txt) is full of $(sourcedir) referencing the python source directory (after a build), e.g., LIBHACL_SHA2_A=Modules/_hacl/libHacl_Hash_SHA2.a. freeze should generate paths as (then) the lib is located by the linux distro.

With Python 3.11.x everythings works like a charm. Should Ubuntu deliver the Module _hacl as binary, or which form? I cannot find any hint in the system and in the package repo. If desired, I could open a respective issue in Ubuntu Lauchpad bugtracker.

Many thanks for looking into it,
Chris

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-116043: Simplify HACL* build #119320

[chris21k]

As far as I can see, building libHacl_Hash_SHA1.a is not necessary. Simply generate directories and emtpy file Modules/_hacl/libHacl_Hash_SHA1.a in the frozen directory, i.e., where make is called. I conjecture that libHacl_Hash_SHA1.a is during python build itself statically linked to _hashlib.cpython-312-x86_64-linux-gnu.so which is dynamically bound to the feeze result executable. Thus, even its hash functions can be used with this trick applied.

If this is correct, imho the generated Makefile by freeze should not include libHacl_Hash_SHA1.a in any way, which should be fixed.

Maybe an easy solution is to not link a binary .a file but the single .o files of the compiled .c files to the .so as done with other hash functions different to sha2. This would probably not fool freeze, then. The former, linking .a looks to me like some hacky proceeding.

Thx,
Chris

[gpshead]

+cc @msprotz as this relates to #99108 code.

[msprotz]

@gpshead I think you added the support for the .a file in an early version where sha2 was scattered across multiple C files, but I have not resorted to that mechanism for the other hash algorithms -- if that helps, I can submit a PR that gets rid of the .a in favor of a single .o?

[picnixz]

We fixed this for 3.14+ only as 3.12 is security-only. See also 131298.