GH-132983: Support finding libzstd without pkg-config

[emmatyping]

GH-133479 removed the logic to check for libzstd outside of pkg-config. This commit adds that logic back with a check for the version so that users can provide their own libzstd. This is to bring parity with lzma, bz2, and zlib detection.

The issues previously seen on RHEL 8 were due to how the checks are ordered. After #133479, if a library version was too old, the PKG_CHECK_MODULES macro would take the "not found" path. This would then search for and find zstd.h and a libzstd with the symbol ZDICT_finalizeDictionary, but then compilation would fail when the header check occured. #133502 removed this path solving the issue on RHEL 8. However, this also make it impossible for a user to set a custom build of libzstd as the dependency (as they can do for liblzma, libbz2, etc.).

In this PR we add this path back, but add an additional check for the version of the libraries. If a user/third party packager would like to point to their own libzstd, they can do so. We avoid the issue #133479 was meaning to solve by ensuring that the version check only runs if we already have the zstd.h header and libzstd library found.

• Issue: Implement PEP 784 - Adding Zstandard to the Python standard library #132983

[emmatyping]

!buildbot RHEL8

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @emmatyping for commit f3dd7f7 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133550%2Fmerge

The command will test the builders whose names match following regular expression: RHEL8

The builders matched are:

• PPC64LE RHEL8 LTO PR
• PPC64LE RHEL8 Refleaks PR
• AMD64 RHEL8 LTO + PGO PR
• aarch64 RHEL8 PR
• aarch64 RHEL8 Refleaks PR
• AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
• AMD64 RHEL8 PR
• aarch64 RHEL8 LTO + PGO PR
• aarch64 RHEL8 LTO PR
• s390x RHEL8 Refleaks PR
• PPC64LE RHEL8 LTO + PGO PR
• s390x RHEL8 PR
• s390x RHEL8 LTO PR
• s390x RHEL8 LTO + PGO PR
• AMD64 RHEL8 FIPS No Builtin Hashes PR
• AMD64 RHEL8 Refleaks PR
• AMD64 RHEL8 LTO PR
• PPC64LE RHEL8 PR

[emmatyping]

Scheduled RHEL 8 but I also want to double check the ghcr.io/cirruslabs/macos-runner:sonoma job.

[emmatyping]

Okay this is looking good.
RHEL 8:

... checking for libzstd >= 1.4.5... no checking for zstd.h... no checking for zdict.h... no ... checking for stdlib extension module _zstd... missing ... 

macOS Sonoma:

... checking for libzstd >= 1.4.5... yes ... checking for stdlib extension module _zstd... yes ... 

The AMD64 RHEL8 FIPS No Builtin Hashes PR buildbot failure looks unrelated.

[AA-Turner]

I've opened #133565 as an alternative with slightly different (and I think also faster) autoconf syntax, what do you think?

A

[emmatyping]

!buildbot RHEL8

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @emmatyping for commit 235fe69 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F133550%2Fmerge

The command will test the builders whose names match following regular expression: RHEL8

The builders matched are:

• PPC64LE RHEL8 LTO PR
• PPC64LE RHEL8 Refleaks PR
• AMD64 RHEL8 LTO + PGO PR
• aarch64 RHEL8 PR
• aarch64 RHEL8 Refleaks PR
• AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
• AMD64 RHEL8 PR
• aarch64 RHEL8 LTO + PGO PR
• aarch64 RHEL8 LTO PR
• s390x RHEL8 Refleaks PR
• PPC64LE RHEL8 LTO + PGO PR
• s390x RHEL8 PR
• s390x RHEL8 LTO PR
• s390x RHEL8 LTO + PGO PR
• AMD64 RHEL8 FIPS No Builtin Hashes PR
• AMD64 RHEL8 Refleaks PR
• AMD64 RHEL8 LTO PR
• PPC64LE RHEL8 PR

[erlend-aasland]

Thanks, this seems correct to me! It would be unfortunate if we had a different set of configure rules for libzstd than for other third-party dependencies.

[emmatyping]

Closing in favor of #133565