bpo-31783: Fix a race condition creating workers during shutdown

[brianquinlan]

https://bugs.python.org/issue31783

[markshannon]

Can you add a test for this, thanks.

[brianquinlan]

The race condition occurs with the following set of context switches:

Main Thread Thread #1 ========= . ======== 1. ThreadPoolExecutor.submit() ... if _shutdown: <===== 2. thread._python_exit _shutdown = true items = list(_threads_queues.items()) =====> 3. ... _threads_queues[t] = self._work_queue <===== 4. for t, q in items: q.put(None) for t, q in items: t.join() 

At this point there is a new thread added to _threads_queues but thread._python_exit won't join it because it was added to _threads_queuesafter a copy was made (i.e.
items = list(_threads_queues.items()))

The bug has a patch with time.sleep() to force the pathological sequencing and a test script to demonstrate the problem.

[brianquinlan]

I don't think writing a test for this would be feasible because provoking the race condition requires very specific timing that, at least on my machine, I can't provoke without adding sleep calls to the code.

[bedevere-bot]

@brianquinlan: Please replace # with GH- in the commit message next time. Thanks!

[miss-islington]

Thanks @brianquinlan for the PR 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

[miss-islington]

Thanks @brianquinlan for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖 I'm not a witch! I'm not a witch!

[miss-islington]

Sorry, @brianquinlan, I could not cleanly backport this to 3.8 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 242c26f53edb965e9808dd918089e664c0223407 3.8

[miss-islington]

Sorry @brianquinlan, I had trouble checking out the 3.7 backport branch.
Please backport using cherry_picker on command line.
cherry_picker 242c26f53edb965e9808dd918089e664c0223407 3.7