bpo-39511: PyThreadState_Clear() calls on_delete

[vstinner]

PyThreadState.on_delete is a callback used to notify Python when a
thread completes. _thread._set_sentinel() function creates a lock
which is released when the thread completes. It sets on_delete
callback to the internal release_sentinel() function. This lock is
known as Threading._tstate_lock in the threading module.

The release_sentinel() function uses the Python C API. The problem is
that on_delete is called late in the Python finalization, when the C
API is no longer fully working.

The PyThreadState_Clear() function is now responsible to call
PyThreadState.on_delete callback. Previously, PyThreadState_Delete()
was responsible for that.

The release_sentinel() function is now called when the C API is still
fully working.

https://bugs.python.org/issue39511

[vstinner]

Eric snow @ericsnowcurrently already faced this issue when trying to fix https://bugs.python.org/issue36854 with his PR #13219. I managed to work around this issue by clearing the Python thread state later: commit 9da7430.

But to implement https://bugs.python.org/issue39511 correctly (clear singletons at exit), I have to fix this issue.

[vstinner]

I decided to not document the change in What's New in Python 3.9 since I don't think that PyThreadState_Clear() should be called directly. By the way, I'm not sure why it's documented at all nor why it's a public function. For me, it belongs more to the internal C API.

[ericsnowcurrently]

LGTM

I only have one suggestion on the wording of the docs.

[ericsnowcurrently]

This usage of "responsible" could be confusing. Consider instead:

 This function now calls the :c:member:`PyThreadState.on_delete` callback. Previously, that happened in :c:func:`PyThreadState_Delete`. 

[vstinner]

Thanks, I updated the doc.