gh-108388: Convert test_concurrent_futures to package

[vstinner]

Convert test_concurrent_futures to a package of 7 sub-tests.

Add remote_globals to create_executor_tests()

• Issue: Split long running test suites up into smaller sub suites to reduce our regrtest long tail on multi-core systems. #108388

[vstinner]

Timing of running tests: ./python -m test test_concurrent_futures -j0 --slowest on a machine with 12 threads / 6 CPU cores.

• reference: 2 min 43 sec
• PR: 1 min 13 sec

With the PR:

10 slowest tests: - test_concurrent_futures.test_wait: 1 min 13 sec - test_concurrent_futures.test_process_pool: 42.6 sec - test_concurrent_futures.test_shutdown: 20.1 sec - test_concurrent_futures.test_as_completed: 12.0 sec - test_concurrent_futures.test_deadlock: 11.1 sec - test_concurrent_futures.test_future: 3.2 sec - test_concurrent_futures.test_init: 2.3 sec 

[sobolevn]

To fix the CI

[vstinner]

GHA job timings:

• Windows x86: 24 min 37 sec
• Windows x64: 19 min 27 sec
• macOS: 18 min 22 sec
• Ubuntu: 10 min
• Address Sanitizer: 14 min 49 sec

Details.

Windows x86:

10 slowest tests: - test_importlib: 3 min 39 sec - test.test_multiprocessing_spawn.test_manager: 2 min 49 sec - test_compileall: 2 min 47 sec - test_venv: 2 min 33 sec - test_tarfile: 2 min 28 sec - test.test_multiprocessing_spawn.test_processes: 2 min 10 sec - test_threading: 2 min 2 sec - test_fstring: 1 min 56 sec - test_queue: 1 min 52 sec - test_socket: 1 min 43 sec 

Windows x64:

10 slowest tests: - test.test_multiprocessing_spawn.test_processes: 3 min 18 sec - test.test_multiprocessing_spawn.test_manager: 2 min 36 sec - test_regrtest: 2 min 27 sec - test_importlib: 2 min 6 sec - test_compileall: 2 min 5 sec - test_venv: 2 min 2 sec - test_zipfile: 1 min 57 sec - test_io: 1 min 47 sec - test_tarfile: 1 min 45 sec - test_largefile: 1 min 38 sec 

macOS:

10 slowest tests: - test.test_multiprocessing_spawn.test_processes: 3 min 20 sec - test_tarfile: 2 min 48 sec - test.test_multiprocessing_forkserver.test_processes: 2 min 19 sec - test_pickle: 1 min 58 sec - test_cppext: 1 min 40 sec - test_statistics: 1 min 39 sec - test_compileall: 1 min 25 sec - test.test_concurrent_futures.test_wait: 1 min 24 sec - test_subprocess: 1 min 22 sec - test.test_multiprocessing_spawn.test_misc: 1 min 21 sec 

Ubuntu:

10 slowest tests: - test_gdb: 2 min 23 sec - test.test_multiprocessing_spawn.test_processes: 1 min 31 sec - test.test_concurrent_futures.test_wait: 1 min 22 sec - test_signal: 1 min 2 sec - test.test_multiprocessing_forkserver.test_processes: 1 min 1 sec - test_subprocess: 1 min - test_weakref: 44.6 sec - test_venv: 44.2 sec - test_socket: 42.1 sec - test.test_concurrent_futures.test_process_pool: 41.4 sec 

Address Sanitizer:

10 slowest tests: - test_subprocess: 2 min 11 sec - test_compileall: 2 min 9 sec - test_venv: 2 min - test_gdb: 1 min 39 sec - test_weakref: 1 min 32 sec - test_tarfile: 1 min 26 sec - test_regrtest: 1 min 12 sec - test_source_encoding: 1 min 11 sec - test_multiprocessing_main_handling: 1 min 1 sec - test_decimal: 1 min 

[gpshead]

I believe you unintentionally got rid of most of the Threading tests.

(do not merge this even after fixing up this review round's comments, this requires further very close side by side review that Github's UI is incapable of providing as it lacks the ability to track file splits and moves as a diff of what's changed -- I'll need to spend time on the command line in a client looking things over)

Otherwise: Thank you for taking this on! This test suite has needed refactoring into something manageable forever.

[bedevere-bot]

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

[gpshead]

I moved this PR to Draft state to signal that I don't want anyone to think it is ready to merge even if someone approves it until after we've taken a close look to verify that we haven't lost anything in the refactoring.

[vstinner]

I believe you unintentionally got rid of most of the Threading tests.

I compared ./python -m test test_concurrent_futures --list-cases|wc -l with and without my change: I get 248 test cases in both cases.

[vstinner]

For your comments like "ThreadPoolExecutorTest is defined above but missing here" on test_process_pool.py, I suggest you looking at which tests are executed rather than only looking at the code. For example:

$ ./python -m test test_concurrent_futures --list-cases|grep ThreadPoolExecutorTest test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_default_workers test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_executor_map_current_future_cancel test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_free_reference test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_hang_global_shutdown_lock test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_idle_thread_reuse test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map_exception test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map_submits_without_iteration test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_map_timeout test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_max_workers_negative test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_no_stale_references test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_saturation test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_shutdown_race_issue12456 test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_submit test.test_concurrent_futures.test_process_pool.ThreadPoolExecutorTest.test_submit_keyword 

You can compare between the main branch and my PR: I get the same 15 tests, but with my PR, test names also get an additional test_process_pool. in their name.

[vstinner]

I splitted test_process_pool in two testrs:

• test_thread_pool: 15 tests, takes 10.6 sec
• test_process_pool: 60 tests, takes 32.0 sec

Before, test_process_poll was the sum (75 tests, ~42 sec) 🙂.

[gpshead]

Thanks. confirmed locally as well, all tests are present and accounted for!

[vstinner]

Thanks for the review @gpshead! For sure, there is room for improvement for these tests. I hope that this split will ease future maintenance.

[miss-islington]

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[miss-islington]

Sorry, @vstinner, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker aa6f787faa4bc45006da4dc2f942fb9b82c98836 3.12

[bedevere-bot]

GH-108443 is a backport of this pull request to the 3.12 branch.

[gpshead]

the merge conflict on this is probably just the list in regrtest runtests.py. retry that automation after the test_multiprocessing one is in 3.12.

[vstinner]

the merge conflict on this is probably just the list in regrtest runtests.py. retry that automation after the test_multiprocessing one is in 3.12.

Right. I backported the change to Python 3.12. There is a minor conflict in libregrtest on:

 SPLITTESTDIRS ={"test_asyncio", + "test_concurrent_futures", } 

The backport is in conflict with PR #108401 (split multiprocessing tests) which may be merged first.

[bedevere-app]

GH-109704 is a backport of this pull request to the 3.11 branch.

[vstinner]

Since I was asked to backport all test changes (including refactoring) to stable branches, I backport this change to Python 3.11: GH-109704.