[3.11] gh-108822: Backport libregrtest changes from the main branch#108820
Uh oh!
There was an error while loading. Please reload this page.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
…iled test (python#106169) (python#106175)" This reverts commit d5418e9.
…fails (pythonGH-30895) (pythonGH-103342)" This reverts commit ecb09a8.
…onGH-98492)" This reverts commit b2aa28e.
…rary file (pythonGH-94253) (pythonGH-94408)" This reverts commit 0122ab2.
This reverts commit f0f3a42.
…sues (pythonGH-94002) (pythonGH-94006)" This reverts commit 1073184.
) When running tests with -jN, create a temporary directory per process and mark a test as "environment changed" if a test leaks a temporary file or directory. (cherry picked from commit e566ce5)
(cherry picked from commit 36934a1)
…94002) - Emscripten's default umask is too strict, see emscripten-core/emscripten#17269 - getuid/getgid and geteuid/getegid are stubs that always return 0 (root). Disable effective uid/gid syscalls and fix tests that use chmod() current user. - Cannot drop X bit from directory. (cherry picked from commit 2702e40)
Reindent files which were not properly formatted (PEP 8: 4 spaces). Remove also some trailing spaces. (cherry picked from commit e87ada4)
…onGH-94253) Co-authored-by: Victor Stinner <vstinner@python.org> (cherry picked from commit 199ba23)
…ythonGH-96689) Automerge-Triggered-By: GH:zware (cherry picked from commit 9c8f379)
On Windows, when the Python test suite is run with the -jN option, the ANSI code page is now used as the encoding for the stdout temporary file, rather than using UTF-8 which can lead to decoding errors. (cherry picked from commit ec1f6f5)
…thon#98904) The Python test suite now fails wit exit code 4 if no tests ran. It should help detecting typos in test names and test methods. * Add "EXITCODE_" constants to Lib/test/libregrtest/main.py. * Fix a typo: "NO TEST RUN" becomes "NO TESTS RAN" (cherry picked from commit c76db37)
The Python test runner (libregrtest) now logs Python build information like "debug" vs "release" build, or LTO and PGO optimizations. (cherry picked from commit 3c89202)
… parallel by sharding. (python#103927) This runs test_asyncio sub-tests in parallel using sharding from Cinder. This suite is typically the longest-pole in runs because it is a test package with a lot of further sub-tests otherwise run serially. By breaking out the sub-tests as independent modules we can run a lot more in parallel. After porting we can see the direct impact on a multicore system. Without this change: Running make test is 5 min 26 seconds With this change: Running make test takes 3 min 39 seconds That'll vary based on system and parallelism. On a `-j 4` run similar to what CI and buildbot systems often do, it reduced the overall test suite completion latency by 10%. The drawbacks are that this implementation is hacky and due to the sorting of the tests it obscures when the asyncio tests occur and involves changing CPython test infrastructure but, the wall time saved it is worth it, especially in low-core count CI runs as it pulls a long tail. The win for productivity and reserved CI resource usage is significant. Future tests that deserve to be refactored into split up suites to benefit from are test_concurrent_futures and the way the _test_multiprocessing suite gets run for all start methods. As exposed by passing the -o flag to python -m test to get a list of the 10 longest running tests. --------- Co-authored-by: Carl Meyer <carl@oddbird.net> Co-authored-by: Gregory P. Smith <greg@krypto.org> [Google, LLC] (cherry picked from commit 9e011e7)
Display the sanitizers present in libregrtest. Having this in the CI output for tests with the relevant environment variable displayed will help make it easier to do what we need to create an equivalent local test run. (cherry picked from commit 852348a)
…on#106169) When running the Python test suite with -jN option, if a worker stdout cannot be decoded from the locale encoding report a failed testn so the exitcode is non-zero. (cherry picked from commit 2ac3eec)
) * Cleanup libregrtest code logging ASAN_OPTIONS. * Fix a typo on "ASAN_OPTIONS" vs "MSAN_OPTIONS". (cherry picked from commit 3a1ac87)
Currently, test_asyncio package is only splitted into sub-tests when using command "./python -m test". With this change, it's also splitted when passing it on the command line: "./python -m test test_asyncio". Remove the concept of "STDTESTS". Python is now mature enough to not have to bother with that anymore. Removing STDTESTS simplify the code. (cherry picked from commit 174e9da)
test_netrc, test_pep646_syntax and test_xml_etree now return results in the test_main() function. Changes: * Rewrite TestResult as a dataclass with a new State class. * Add test.support.TestStats class and Regrtest.stats_dict attribute. * libregrtest.runtest functions now modify a TestResult instance in-place. * libregrtest summary lists the number of run tests and skipped tests, and denied resources. * Add TestResult.has_meaningful_duration() method. * Compute TestResult duration in the upper function. * Use time.perf_counter() instead of time.monotonic(). * Regrtest: rename 'resource_denieds' attribute to 'resource_denied'. * Rename CHILD_ERROR to MULTIPROCESSING_ERROR. * Use match/case syntadx to have different code depending on the test state. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> (cherry picked from commit d4e534c)
MemberAuthor
vstinner commented Sep 2, 2023
I had to revert multiple changes and cherry-pick them again to reduce the number of merge conflicts. This PR adds 5 NEWS entries. regrtest now checks if a test leaks temporary files or directories if run On Windows, when the Python test suite is run with the The Python test suite now fails wit exit code 4 if no tests ran. It should The Python test runner (libregrtest) now logs Python build information like When running the Python test suite with |
vstinner changed the title MemberAuthor
vstinner commented Sep 2, 2023
I will wait until the NEW entry is added by #108821 to also include it in this 3.11 backport. |
MemberAuthor
vstinner commented Sep 2, 2023
Ok, I added the Changelog entry for statistics: |
MemberAuthor
vstinner commented Sep 2, 2023
This change is related the issue #93353. It was a bug triggered by pip fixed in May 2022 by fixing importlib.resources._tempfile() finalizer: commit dc6dd8e (backport to 3.11). The issue was also reported to pip: issue pypa/pip#10753. Python 3.11 test suite should not leak temporary files. It should be fine to backport this new check. |
Member
serhiy-storchaka commented Sep 3, 2023
Wow, wow, WOW! |
serhiy-storchaka approved these changes Sep 3, 2023
Member
serhiy-storchaka left a comment
serhiy-storchaka left a comment There was a problem hiding this comment.
LGTM.
You cannot believe that I reviewed all these changes. I did not. I compared the result with the main branch.
MemberAuthor
vstinner commented Sep 3, 2023
Merged, thanks for the review. |
MemberAuthor
vstinner commented Sep 3, 2023
Sure, I would do the same :-) I also trust |
bedevere-bot commented Sep 3, 2023
|
MemberAuthor
vstinner commented Sep 3, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
setUp[Class|Module]fails (bpo-46523: fix tests rerun whensetUp[Class|Module]fails #30895)setUp[Class|Module]fails (bpo-46523: fix tests rerun whensetUp[Class|Module]fails #30895) ([3.11] bpo-46523: fix tests rerun whensetUp[Class|Module]fails (GH-30895) #103342)"