gh-91349: Replace zlib with zlib-ng in Windows build#131438

zooba · 2025-03-19T01:11:08Z

Issue: Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349

zooba · 2025-03-19T01:12:37Z

Misc/externals.spdx.json

 {
 "referenceCategory": "SECURITY",
-"referenceLocator": "cpe:2.3:a:zlib:zlib:1.3.1:*:*:*:*:*:*:*",
+"referenceLocator": "cpe:2.3:a:zlib-ng:zlib-ng:2.2.4:*:*:*:*:*:*:*",


I'll admit I just guessed this, in order to unblock my testing (couldn't build at all with an invalid SBOM). If it's not right, let me know

This is likely the CPE that would get used, although we can't be for certain until a CVE exists with it... the only one I could find with some searching is cpe:...:zlib-ng:minizip-ng which is for a different component but at least the organization is a match.

zooba · 2025-03-19T01:13:14Z

Modules/zlibmodule.c

 return-1;
 }
+#ifdefZLIBNG_VERSION
+if (PyModule_Add(mod, "ZLIBNG_VERSION",


Figured we'd want some way to detect this other than looking at the text in ZLIB_VERSION

makes sense. annoying to have optional attributes that one would use hasattr or getattr on instead of blindly accessing, but realistically nobody should care as this is more internal informational so this is fine.

Yeah, the main place it'll be used is in pythoninfo, which already handles absent attributes.
We have more offensive optional attributes in our extension modules 😆

zooba · 2025-03-19T01:24:45Z

I've reviewed the three header files added to PC and they seem generic. It's a shame they aren't part of the sources, but figured it's easier/safer to just copy them into our source tree so that the cpython-source-deps repo is a straight import.

bedevere-bot · 2025-03-19T01:35:18Z

🤖 New build scheduled with the buildbot fleet by @zooba for commit 08eecb1 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F131438%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

PCbuild/pythoncore.vcxproj

zooba · 2025-03-19T13:45:34Z

The two failing buildbots appear unrelated, and everything else passed. All I changed in the last commit was docs, so I'm not going to rerun the buildbots unless someone thinks it's necessary or worthwhile.

sethmlarson

SBOM changes LGTM!

sethmlarson · 2025-03-19T13:38:26Z

Misc/externals.spdx.json

 {
 "referenceCategory": "SECURITY",
-"referenceLocator": "cpe:2.3:a:zlib:zlib:1.3.1:*:*:*:*:*:*:*",
+"referenceLocator": "cpe:2.3:a:zlib-ng:zlib-ng:2.2.4:*:*:*:*:*:*:*",


This is likely the CPE that would get used, although we can't be for certain until a CVE exists with it... the only one I could find with some searching is cpe:...:zlib-ng:minizip-ng which is for a different component but at least the organization is a match.

mdboom · 2025-03-19T14:22:55Z

I'm kicking off a pyperformance benchmarking run with this -- we have a non-zero usage of zlib in that suite.

mdboom · 2025-03-19T18:02:38Z

The results for this on pyperformance are basically within the noise / no change. Not all that surprising, given that there are no benchmarks specifically aimed at zlib. https://github.com/faster-cpython/benchmarking-public/tree/main/results/bm-20250319-3.14.0a6+-548daa7

gpshead · 2025-03-19T18:21:32Z

Modules/zlibmodule.c

 return-1;
 }
+#ifdefZLIBNG_VERSION
+if (PyModule_Add(mod, "ZLIBNG_VERSION",


makes sense. annoying to have optional attributes that one would use hasattr or getattr on instead of blindly accessing, but realistically nobody should care as this is more internal informational so this is fine.

chris-eibl · 2025-03-20T16:57:13Z

This has broken the tailcall builds on Windows: https://github.com/python/cpython/actions/runs/13967899073/job/39102464260

D:\a\cpython\cpython\externals\\zlib-ng-2.2.4\\functable.c(79,26): error : use of undeclared identifier 'slide_hash_sse2'; did you mean 'slide_hash_c'? [D:\a\cpython\cpython\PCbuild\zlib-ng.vcxproj] D:\a\cpython\cpython\externals\\zlib-ng-2.2.4\\functable.c(123,26): error : use of undeclared identifier 'slide_hash_avx2'; did you mean 'slide_hash_c'? [D:\a\cpython\cpython\PCbuild\zlib-ng.vcxproj]

zooba · 2025-03-20T17:01:30Z

Probably needs some fixed preprocessor checks (hopefully only in what's in the PC directory) to better handle the intrinsics being present/absent.

Though I notice that build warns NOTE: Visual Studio not detected. LLVM does not provide a C/C++ standard library and may be unable to locate MSVC headers., which might also be relevant.

chris-eibl · 2025-03-20T17:24:07Z

I think, this is because here
https://github.com/zlib-ng/zlib-ng/blob/860e4cff7917d93f54f5d7f0bc1d0e8b1a3cb988/fallback_builtins.h#L4-L25
in line 4 __clang__ gets excluded and hence #define HAVE_BUILTIN_CTZ is not set (but in the MSVC case it is).

Then, in
https://github.com/zlib-ng/zlib-ng/blob/860e4cff7917d93f54f5d7f0bc1d0e8b1a3cb988/arch/x86/x86_functions.h#L13-L20
we dont get the prototype for slide_hash_sse2 in line 17, hence the error.

Sorry, I cannot get the permalinks to work here the usual way I am doing it :(

chris-eibl · 2025-03-20T17:30:51Z

Though I notice that build warns NOTE: Visual Studio not detected. LLVM does not provide a C/C++ standard library and may be unable to locate MSVC headers., which might also be relevant.

FWIW, when building with the bundled clang-cl of VS I do not have these warnings, but get the same error.

zooba · 2025-03-20T17:37:20Z

Most likely zlib-ng would generate different versions of the header files I checked into the PC directory for Clang. Figuring out what differences belong in there and detecting it dynamically rather than using CMake is the way forward.

This should get a new issue (because clang-cl isn't a supported configuration, so we don't have to revert this change until it's working). Tag me on it and we can continue there.

chris-eibl · 2025-03-20T17:44:41Z

PCbuild/zlib-ng.vcxproj

+ <PrecompiledHeader>NotUsing</PrecompiledHeader>
+ <AdditionalIncludeDirectories>$(zlibNgDir);$(PySourceDir)PC;$(GeneratedZlibNgDir);%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+ <PreprocessorDefinitions>%(PreprocessorDefinitions);ZLIB_COMPAT;WITH_GZFILEOP;NO_FSEEKO;HAVE_BUILTIN_ASSUME_ALIGNED;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;</PreprocessorDefinitions>
+ <PreprocessorDefinitionsCondition="$(Platform) == 'Win32' or $(Platform) == 'x64'">%(PreprocessorDefinitions);X86_FEATURES;X86_HAVE_XSAVE_INTRIN;X86_SSE2;X86_SSSE3;X86_SSE42;X86_PCLMULQDQ_CRC;X86_AVX2;X86_AVX512;X86_AVX512VNNI;X86_VPCLMULQDQ_CRC</PreprocessorDefinitions>


Does this mean that for all executables X86_AVX512 code will be emitted and used?
See e.g. adler32_avx512.c, where the whole file is guarded with X86_AVX512 and there is code like __m512i in it.

I believe it's detected and called dynamically based on availability, but it's all compiled at compile time.

Yeah, I hope so, too.

chris-eibl · 2025-03-20T20:34:53Z

PCbuild/zlib-ng.vcxproj

+ <PreprocessorDefinitions>%(PreprocessorDefinitions);ZLIB_COMPAT;WITH_GZFILEOP;NO_FSEEKO;HAVE_BUILTIN_ASSUME_ALIGNED;_CRT_SECURE_NO_DEPRECATE;_CRT_NONSTDC_NO_DEPRECATE;</PreprocessorDefinitions>
+ <PreprocessorDefinitionsCondition="$(Platform) == 'Win32' or $(Platform) == 'x64'">%(PreprocessorDefinitions);X86_FEATURES;X86_HAVE_XSAVE_INTRIN;X86_SSE2;X86_SSSE3;X86_SSE42;X86_PCLMULQDQ_CRC;X86_AVX2;X86_AVX512;X86_AVX512VNNI;X86_VPCLMULQDQ_CRC</PreprocessorDefinitions>
+ <PreprocessorDefinitionsCondition="$(Configuration) == 'Debug'">%(PreprocessorDefinitions);ZLIB_DEBUG</PreprocessorDefinitions>
+ <EnableEnhancedInstructionSetCondition="$(Platform) == 'Win32' or $(Platform) == 'x64'">AdvancedVectorExtensions2</EnableEnhancedInstructionSet>


Unconditionally letting the compiler generate code up to AVX2 for all *.c files here might result in the binary not running on all CPUs?

The big changes here are: - Switching to zlib-ng on Windows (python/cpython#131438) - Using hmac for hashing functions (python/cpython#130157) --------- Co-authored-by: Geoffrey Thomas <geofft@ldpreload.com>

zooba added 4 commits March 19, 2025 00:56

pythongh-91349: Replace zlib with zlib-ng in Windows build
de56164

Updated readme.txt
1347c50

Fix includes for PGO build
7fd8f26

Use correct include dirs
f919749

zooba requested a review from gpshead March 19, 2025 01:11

zooba requested review from a team and sethmlarson as code owners March 19, 2025 01:11

bedevere-appbot added the awaiting core review label Mar 19, 2025

bedevere-appbot mentioned this pull request Mar 19, 2025
Use zlib-ng (fast!) rather than mainline stale zlib in binary releases #91349
Open

Add NEWS
fb6500d

zooba commented Mar 19, 2025
View reviewed changes

zooba added 2 commits March 19, 2025 01:25

Fix debug reference and some warnings
89d44a1

Exclude Intel optimizations better on ARM64
08eecb1

zooba added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 19, 2025

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Mar 19, 2025

rhpvorderman reviewed Mar 19, 2025
View reviewed changes

PCbuild/pythoncore.vcxprojShow resolvedHide resolved

Update docs and whatsnew
548daa7

sethmlarson approved these changes Mar 19, 2025
View reviewed changes

Remove unresolved reference from docs
f108044

gpshead approved these changes Mar 19, 2025
View reviewed changes

bedevere-appbot added awaiting merge and removed awaiting core review labels Mar 19, 2025

Include ZLIBNG_VERSION in pythoninfo
085fa10

zooba enabled auto-merge (squash) March 19, 2025 18:38

zooba merged commit 63a638c into python:mainMar 19, 2025
42 checks passed

bedevere-appbot removed the awaiting merge label Mar 19, 2025

zooba deleted the gh-91349 branch March 19, 2025 19:25

chris-eibl reviewed Mar 20, 2025
View reviewed changes

chris-eibl mentioned this pull request Mar 20, 2025
clang-cl on Windows fails to compile after zlib-ng #131521
Closed

chris-eibl reviewed Mar 20, 2025
View reviewed changes

hugovk mentioned this pull request Mar 21, 2025
Consider distributing zlib-ng astral-sh/python-build-standalone#464
Open

chris-eibl mentioned this pull request Mar 22, 2025
GH-131521: fix clangcl build on Windows for zlib-ng #131526
Merged

zanieb mentioned this pull request Apr 9, 2025
Bump to 3.14.0a7 astral-sh/python-build-standalone#580
Merged

andersonhc mentioned this pull request Oct 15, 2025
Add Python 3.14 support py-pdf/fpdf2#1622
Merged
4 tasks

Uh oh!

gh-91349: Replace zlib with zlib-ng in Windows build#131438

gh-91349: Replace zlib with zlib-ng in Windows build #131438

Uh oh!

Conversation

zooba commented Mar 19, 2025• edited by bedevere-app botLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zooba commented Mar 19, 2025

Uh oh!

bedevere-bot commented Mar 19, 2025

Uh oh!

Uh oh!

zooba commented Mar 19, 2025

Uh oh!

sethmlarson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdboom commented Mar 19, 2025

Uh oh!

mdboom commented Mar 19, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chris-eibl commented Mar 20, 2025

Uh oh!

zooba commented Mar 20, 2025

Uh oh!

chris-eibl commented Mar 20, 2025

Uh oh!

chris-eibl commented Mar 20, 2025

Uh oh!

zooba commented Mar 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

zooba commented Mar 19, 2025•
edited by bedevere-app bot
Loading