gh-107219: Fix concurrent.futures terminate_broken()

[vstinner]

Fix a race condition in _ExecutorManagerThread.terminate_broken(): ignore the InvalidStateError on future.set_exception(). It can happen if the future is cancelled before the caller.

Moreover, test_crash_big_data() now waits explicitly until the executor completes.

• Issue: test_concurrent_futures.test_deadlock: test_crash_big_data() hangs randomly on Windows #107219

[vstinner]

This change doesn't fix the most annoying bug of issue #107219. test_crash_big_data() still hangs on Windows. But it fix this error:

test_crash_big_data (test.test_concurrent_futures.ProcessPoolSpawnExecutorDeadlockTest.test_crash_big_data) ... Warning -- Uncaugh t thread exception: InvalidStateError Exception in thread Thread-9: Traceback (most recent call last): File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\threading.py", line 1059, in _bootstrap_inner self.run() File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\process.py", line 344, in run self.terminate_broken(cause) File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\process.py", line 492, in terminate_broken work_item.future.set_exception(bpe) File "C:\Users\KIRILL-1\CLionProjects\cpython\Lib\concurrent\futures\_base.py", line 559, in set_exception raise InvalidStateError('{}:{!r}'.format(self._state, self)) concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x2cf561cf260 state=cancelled> 0.55s Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2) Warning -- Dangling thread: <_MainThread(MainThread, started 11880)> Warning -- Dangling thread: <Thread(QueueFeederThread, started daemon 4112)> ok 

link: #107219 (comment)

[vstinner]

cc @pitrou

[vstinner]

This change doesn't fix the most annoying bug of issue #107219. test_crash_big_data() still hangs on Windows.

If it wasn't obvious enough, the test decided to hang on Windows x64 CI :-)

0:41:15 load avg: 0.02 [462/462/3] test.test_concurrent_futures.test_deadlock process crashed (Exit code 1) Timeout (0:20:00)! Thread 0x00000c90 (most recent call first): File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 282 in _send_bytes File "D:\a\cpython\cpython\Lib\multiprocessing\connection.py", line 199 in send_bytes File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 246 in _feed File "D:\a\cpython\cpython\Lib\threading.py", line 996 in run File "D:\a\cpython\cpython\Lib\threading.py", line 1059 in _bootstrap_inner File "D:\a\cpython\cpython\Lib\threading.py", line 1016 in _bootstrap Thread 0x00001b78 (most recent call first): File "D:\a\cpython\cpython\Lib\threading.py", line 1153 in _wait_for_tstate_lock File "D:\a\cpython\cpython\Lib\threading.py", line 1133 in join File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 199 in _finalize_join File "D:\a\cpython\cpython\Lib\multiprocessing\util.py", line 224 in __call__ File "D:\a\cpython\cpython\Lib\multiprocessing\queues.py", line 151 in join_thread File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 560 in join_executor_internals File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 514 in terminate_broken File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 344 in run File "D:\a\cpython\cpython\Lib\threading.py", line 1059 in _bootstrap_inner File "D:\a\cpython\cpython\Lib\threading.py", line 1016 in _bootstrap Thread 0x000012cc (most recent call first): File "D:\a\cpython\cpython\Lib\threading.py", line 1153 in _wait_for_tstate_lock File "D:\a\cpython\cpython\Lib\threading.py", line 1133 in join File "D:\a\cpython\cpython\Lib\concurrent\futures\process.py", line 843 in shutdown File "D:\a\cpython\cpython\Lib\concurrent\futures\_base.py", line 647 in __exit__ File "D:\a\cpython\cpython\Lib\test\test_concurrent_futures\test_deadlock.py", line 236 in test_crash_big_data