bpo-31901: atexit callbacks should be run at subinterpreter shutdown

[ghost]

This PR moves the atexit callbacks from Python runtime context to interpreter state.
Also, Py_EndInterpreter() now calls atexit callbacks of the interpreter.

These changes make it possible to implement PEP-489 multiphase initialization for atexit module, which is also done in this PR.

https://bugs.python.org/issue31901

[pitrou]

A nit, but it would seem to me more conventional to take the user data (the PyObject *) as second argument.

[pitrou]

Is it actually necessary to use C code for this? As opposed to calling support.run_in_subinterp from test_atexit (as already done in two other tests).

[ghost]

Awesome, didn't know about this thing, it is rewritten in Python now.

[pitrou]

Thank you for the PR! I posted a couple comments.

[pitrou]

There was a crash on AppVeyor. I've retried the bulld in case it's an unrelated sporadic issue (?).

[pitrou]

Looks like the AppVeyor crash isn't sporadic after all (and your PR is the only one to trigger it). Do you have a Windows machine or VM to test?

[pitrou]

By the way, analyzing this PR led me to #4826 :-)

[ghost]

Unfortunately, I don't have any Windows machine.
About #4826, the _Py_PyAtExit should really not be called from anywhere else than atexit module, otherwise, some sort of list instead of single pointer would be needed. Should I have a look at that as a part of this PR?

[pitrou]

PR #4826 was merged, so you just have to rebase/merge this PR on git master (you will have to resolve conflicts, which should be relatively easy here).

[ghost]

And about the Windows crash?

[pitrou]

If it persists after git master is merged in, I'll try to reproduce on my Windows VM (if I manage to unstuck it).

[pitrou]

Ok, I think I've found the culprit for the Windows crash: now that you're using PEP 489 multiphase init support, modstate can be NULL in m_traverse and m_clear (and probably m_free too), so you have to guard against this.

[pitrou]

Note I don't know exactly why that happens, so I asked at https://mail.python.org/pipermail/python-dev/2017-December/151238.html

[ghost]

@pitrou this PR is ready for review

[pitrou]

This looks good to me. I'm gonna merge soon if no other core developer chimes in.

[encukou]

The documentation issue is tracked in bpo-32374.