Use zlib-ng (fast!) rather than mainline stale zlib in binary releases

[gpshead]

BPO	47193
Nosy	@gpshead, @pfmoore, @tjguk, @zware, @zooba, @corona10, @arhadthedev

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-04-01.19:19:24.173>
labels = ['3.11', 'OS-windows', 'performance']
title = 'Use zlib-ng rather than zlib in binary releases'
updated_at = <Date 2022-04-02.09:48:33.061>
user = 'https://github.com/gpshead'

bugs.python.org fields:

activity = <Date 2022-04-02.09:48:33.061>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Windows']
creation = <Date 2022-04-01.19:19:24.173>
creator = 'gregory.p.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 47193
keywords = []
message_count = 1.0
messages = ['416508']
nosy_count = 7.0
nosy_names = ['gregory.p.smith', 'paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'corona10', 'arhadthedev']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue47193'
versions = ['Python 3.11']

Linked PRs

• gh-91349: Replace zlib with zlib-ng in Windows build #131438
• gh-91349: Adjust default compression level to 6 (down from 9) in gzip and tarfile #131470
• gh-91349: Expose the crc32 function from the lzma library #131721

[gpshead]

zlib-ng is an optimized zlib library with better performance on most architectures (with contributions from the likes of Google, Cloudflare, and Intel). It is API compatible with zlib. https://github.com/zlib-ng/zlib-ng

I believe the only platform we don't use the OS's own zlib on is Windows so I'm tagging this issue Windows.

[gpshead]

if this hasn't happened in the 3.11 betas this is presumably bumped to 3.12.

[zooba]

I started taking a look at this and it seems like we can build it without having to worry about their build system by renaming the zconf.h.in and zconf-ng.h.in files to remove the .in, and setting ZLIB_COMPAT preprocessor in pythoncore.vcxproj (as well as referencing the files).

Running test_zlib the only failures seem to be testing for certain failures that no longer fail, but I've got no idea how important they are. Also no indication of the performance impact, or anything else that may change (e.g. new DLL exports, etc.), but it certainly does seem like a feasible drop-in replacement.

[rhpvorderman]

@zooba which failures are these? I have accumulated quite some experience with the zlib/gzip formats due to working on python-isal, bindings for the ISA-L library that speeds up zlib-compatible compression by rewriting the algorithms in x86 Assembly language. It is quite good, but not suitable for a drop-in replacement, hence the python-isal project.

Having said that I'd love to help out with anything zlib related in CPython.

[zooba]

I knew I should've kept better track of the changed error messages 😄

Skimming through the tests, I'm pretty sure test_wbits failed in a few places because we expected some "invalid window size" errors that aren't errors with zlib-ng. I don't think any were critical, but I don't know the usage of the library well enough to be sure (e.g. do people expect these errors and change behaviour? or are they always just developer error and cause changes to just make things work?)