gh-99108: Implement HACL* HMAC

[picnixz]

This supersedes #126359.

Ideally, I'd like this to be part of 3.14 but we only have two alpha releases until the first beta.

Note that the HACL* HMAC implementation does not support truncated SHA-2-512/224 (which is different from SHA-2/224) so we need to either ask HACL* to do it or document it in the hmac module documentation.

For reviews, I would strongly advise review commit by commit instead of the entire file. Each commit should compile separately (assuming preceeding ones are present, maybe except the first few commits are not compiling due to some configure/build I forgot to put it at that time, but otherwise the interface was written incrementally).

cc @msprotz@gpshead

• Issue: Replace built-in hashlib with verified implementations from HACL* #99108

---

📚 Documentation preview 📚: https://cpython-previews--130157.org.readthedocs.build/

[picnixz]

@msprotz I'm not sure if it's an HACL issue or not, but Lib_IntVector_Intrinsics_vec256 in libintvector.h is conditionnally exposed, yet it's unconditionally used in _hacl/Hacl_Streaming_Types.h.

[msprotz]

You are correct -- there is one additional step to do to handle this case. Note that HMAC is the first time we have a file that contains references to vec128/vec256 yet may be compiled on a system that has no such types. (Previously, Blake2b_256 was only ever built if vec256 was known at build-time to exist, and likewise with Blake2s_128.)

One option is to conditional includes, but that leads to other difficulties, such as having to hide cases of the agile hmac state union behind #ifdef, and then propagating #ifdefs everywhere in this file. This is error-prone and a lot of maintenance.

The other option we've done is simply to #define Lib_IntVector_Intrinsics_vec256 void * in case the build determines that HACL_CAN_COMPILE_VEC256 is false (and likewise with vec128). This solves all of the problems above and avoids a complicated include dance.

What I don't understand / remember, though, is why this isn't done directly inside libintvector.h. (We currently do it in our mini-configure for CI purposes and emit those #ifdefs in config.h.)

I'd be curious to see if you could manually patch libintvector.h to define those types to void* (in the #else case of #ifdef HACL_CAN_COMPILE_VEC256, and likewise for VEC128) and then see if we get a good build? I would then be happy to upstream this fix. Thank you!

[picnixz]

I'll wait for #130960 to be merged so that I don't need to re-update the SBOMs etc.

[gpshead]

I'd (1) leave the SIMD detection for a followup PR and (2) keep the pedantic checks for the return code.

similarly, documentation updates we're not clear on the need for can be considered separately.

[msprotz]

Agreed that SIMD can be left for later -- this should be transparent from the point of view of the clients.

[chris-eibl]

LGTM. Thank you @picnixz!

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @picnixz for commit 258aa20 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F130157%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @picnixz for commit 258aa20 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F130157%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-refleak-buildbots label again.

[picnixz]

I expect failures on some FIPS-only build bots but that's fine because the test suite is already failing on them. And some failures on other build bots as well (like aarch64 which is known to be flaky these past days). Once all build bots are fine, I'll merge it (but only on Friday, as I'm leaving in a few hours) [so please don't merge it since I want to write the commit message, TiA]

[picnixz]

Thank you all for the feedback, especially @msprotz for the upstream work! The next step is to enable the SIMD support.