gh-131876: use extern storage for _hashlib helper functions

[picnixz]

• Issue: Unify recognization of message digest names for _hashlib and _hmac #131876

[picnixz]

!buildbot WASM Emscripten

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @picnixz for commit 033ba23 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F137301%2Fmerge

The command will test the builders whose names match following regular expression: WASM Emscripten

The builders matched are:

• WASM Emscripten PR

[picnixz]

Ok, I think the Windows build is now ok. I'm pretty sure the WASM build is still broken because I don't think extern would change much here. For WASI, it appears that _hashlib is not even built (maybe related to #136976?). Locally it seems I can more or less build but it fails with ../../../../Python/emscripten_syscalls.c:221:7: error: use of unknown builtin '__builtin_wasm_ref_is_null_extern' [-Wimplicit-function-declaration] so it might be on my side first. Thus I'll try to get the buildbot pass.

[picnixz]

Ok, so I see the problem. libhashlib.a is indeed included by the hashlib module and by all other HACL* components. It's the same issue as #133042 but the "hacky" fix can be as follows: I just link libhashlib.a to the interpreter directly as there is no OpenSSL or anything else that is needed. For now I'll only patch WASM like this. But I really want to avoid this and find a better way to handle all those cryptographic modules with different "libs".

[zooba]

I don't think extern is even necessary here, since that should be assumed for prototypes. Any .c that includes the internal header will resolve the functions at link time, which is the only time we want it to happen (i.e. not at dynamic load time).

The trickier part is trying to use the same C file in multiple projects, which is notoriously troublesome. The Windows build will handle it okay, because it builds the file multiple times in separate locations with separate settings (for each project that it's compiled into), but the Makefile can sometimes be a problem because it'll typically only build the source file once for all projects. Seems to be okay here, but something to be prepared for if reports start coming in that it isn't working.

[picnixz]

!buildbot WASM Emscripten

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @picnixz for commit 561f4a9 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F137301%2Fmerge

The command will test the builders whose names match following regular expression: WASM Emscripten

The builders matched are:

• WASM Emscripten PR

[picnixz]

Urgh, it still doesn't work on Emscripten. Unfortunately, I can't just disable those modules... So I'll need to investigate more.

[picnixz]

Ok, so the issue is that the .a is present in both LOCALMODLIBS because of MODULE__MD5_LDFLAGS and when building _md5 itself (because I need it). I don't really know what to do here. Should I just remove it from MODULE__MD5_LDFLAGS and put it in LIBPYTHON directly?

@hoodmane how can I make an internal CPython-only library that:

• is shared across multiple modules (_hashlib and _md5 etc),
• can in the future be linked with other external libraries (which is why I don't really want to put it directly into all LDFLAGS of all modules), and
• where the modules can be optionally included (that is, we could disable all built-in modules, but we would still need that simple library; similarly, we could enable all those built-in modules, but we only need one copy of that library).

If this is critically hindering the development on WASM (which I believe it is as it's no more possible to compile Python I think?), I will revert my commit (unless the fix I suggest now works, idk).

EDIT: My latest fix is trying to add explicitly the libs that are needed to the interpreter.

[picnixz]

Ok, the latest idea doesn't seem to really work (either I messed up somewhere else or it's fundamentally wrong). I'm going to revert this: #137307 because it's hindering the development of WASM.

[hoodmane]

Well if the archive isn't needed at all the linker will just leave it out. So you could put in LIBPYTHON directly. If you're concerned about it not existing at all you could check if it's completely absent and make an empty .a file in that case? It's a bit convoluted.

Or we could try wrapping the wasm linker and dropping the duplicated library.

[picnixz]

If you're concerned about it not existing at all you could check if it's completely absent and make an empty .a file in that case? It's a bit convoluted.

That... would be interesting indeed.. but ideally, I'd like to have the build process for WASI/WASM more or less the same as for other platforms. It becomes very hard to make incremental changes.

For now, I'll revert the changes so to avoid blocking further development. Another temporary solution is to simply put everything as a static inline function. I'm still not ready for having a common interface for hashlib and hmac for querying available hash functions so we can live with header files only... Since I'm actually pretty busy for the next two weeks, it's better that I revert this and try to find a better alternative (at the same time, I think I'll be able to fix the issue we had with HACL* static libs where I just disabled the modules...)

Or we could try wrapping the wasm linker and dropping the duplicated library.

That was my idea, but we need to expand makefile variables before because the duplication appears because of LDFLAGS being added to MODLIBS. So there are ldflags coming from all cryptographic modules, and those contain .a. My latest attempt was to separate between the values passed to MODLIBS and those passed to the Makefile rule but I failed.

[picnixz]

Hum... apparently my approach here works... I'll cleanup those hacks tomorrow.

[picnixz]

Too hard to solve the conflicts, I'm going to start a fresh PR.