Skip to content

Conversation

@gpshead
Copy link
Member

@gpsheadgpshead commented Feb 14, 2023

A followup to #101707 which broke some builds. This builds HACL* as a library in one place.

This doesn't solve all the problems, the wasm-wasi build is fixed by this. The wasm-enscripten build is not. Because it has a linker that can't cope with the same .a file being listed twice on its command line and decides to ignorantly process that as two inputs with duplicate symbols.

@gpshead
Copy link
MemberAuthor

!buildbot wasm

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 3c7e18e 🤖

The command will test the builders whose names match following regular expression: wasm

The builders matched are:

  • wasm32-wasi PR
  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten browser (dynamic linking, no tests) PR

@gpshead
Copy link
MemberAuthor

The wasm-wasi bot using llvm-ar to build libpython3.12.a appears happy with this.
The wasm-enscripten ones that use emscripten/emar to build libpython3.12.a still does not.

is this the problem? the .a file is showing up on the emcc linking command line twice, is that really the source of the "duplicate" symbols? is emcc that dumb?
@gpshead
Copy link
MemberAuthor

!buildbot wasm

@bedevere-bot
Copy link

🤖 New build scheduled with the buildbot fleet by @gpshead for commit a5bb63e 🤖

The command will test the builders whose names match following regular expression: wasm

The builders matched are:

  • wasm32-wasi PR
  • wasm32-emscripten node (dynamic linking) PR
  • wasm32-emscripten node (pthreads) PR
  • wasm32-emscripten browser (dynamic linking, no tests) PR

@gpshead
Copy link
MemberAuthor

Emscripten emcc which calls wasm-ld winds up with a duplicate of -LModules/_hacl -lHacl_Streaming_SHA2 on its command line or a direct Modules/_hacl/libHacl_Streaming_SHA2.a on its command line (the emcc script turns the first into the latter regardless). And whatever version of enscripten wasm-ld this is... is brain dead and does not deduplicate input files. Thus reporting "duplicate symbol" because the same .a is listed twice. 💩

Real linkers (like clang and thus llvm's ld.lld) do not have a problem with this.

@gpsheadgpshead changed the title Build the hashlib HACL* code as a static library.gh:99108: Build the hashlib HACL* code as a static library.Feb 14, 2023
@gpsheadgpshead changed the title gh:99108: Build the hashlib HACL* code as a static library.gh-99108: Build the hashlib HACL* code as a static library.Feb 14, 2023
@gpsheadgpshead marked this pull request as ready for review February 14, 2023 22:44
@gpshead
Copy link
MemberAuthor

Realistically: We should probably just combine _sha256 and _sha512 into a single _sha2 module as another way to work around this. The .so files for each of those are carrying the code for both regardless given how we link the shared libraries.

@gpsheadgpshead changed the title gh-99108: Build the hashlib HACL* code as a static library.gh-99108: Build the hashlib HACL* code as a static library. (fix wasm builds)Feb 14, 2023
@gpshead
Copy link
MemberAuthor

Realistically: We should probably just combine _sha256 and _sha512 into a single _sha2 module as another way to work around this. The .so files for each of those are carrying the code for both regardless given how we link the shared libraries.

I'll do that as its own follow on PR. This one at least addresses 2 of the 4 bot failures and defines the static library as we want regardless.

@gpsheadgpshead merged commit d777790 into python:mainFeb 14, 2023
@gpsheadgpshead deleted the build/libhacl branch February 14, 2023 23:57
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@gpshead@bedevere-bot