gh-132413: Fix crash in _datetime when used at shutdown (alt)

[neonene]

Create a module when it is not importable.

This patch gives the interpreter state struct an additional pointer to the _datetime module state. Currently, the module state can be reached via a weak reference to the module, which seems problematic because the weakref counts the active module as invalid earlier at shutdown.

(No different than before logically, but the attempt would need to be withdrawn if there should be a reasonable case that the introduced pointer becomes a dangling pointer or one to a reallocated region. #132599 is an alternative.)

(This does not suppress ImportError raised when running the strptime methods at shutdown, which is almost traditional.)

• Issue: SIGSEV in datetime.timedelta (possibly from datetime's C delta_new) #132413

[python-cla-bot]

All commit authors signed the Contributor License Agreement.

[ZeroIntensity]

I don't think we should be storing more things in the interpreter dictionary. It's relatively slow, and at a glance, probably a thread-safety issue in the way it's being used here.

The better solution would be to drop the interpreter dictionary usage here entirely, and just store the module state on the interpreter state.

cc @ericsnowcurrently

[neonene]

It sounds like the concern is not about the issue that needs to be fixed even on 3.13. The current module state access may not be so fast, but I doubt this PR makes it slower. My question for alternatives is that how much we need to care about the thread-safety after finalize_modules() finishes in pylifecycle.c.

[ZeroIntensity]

I meant an existing thread-safety issue, but you're right, let's not address that here. _datetime is a special module, it's fine to make it special on the interpreter state.

• It's significantly less complex to do this through the interpreter state.
• Manual invocation of PyModule_FromDefAndSpec/PyModule_ExecDef like this is very likely going to introduce other bugs. I don't think we have any tests for either of those functions, let alone during finalization.

[neonene]

I agree to the risk of manual invocation during finalization.

Two versions work fine:

• This PR: ABI changed. Module state is added to the interpreter state.

• #132599: ABI not changed. Backportable.

Both require the interpreter dict. I failed to add a module object to the interpreter state (bad reloading).

[ZeroIntensity]

The interpreter state isn't part of the public ABI, we can (and do) change it in backports.

[ZeroIntensity]

Let me know when you're ready for a re-review.

[neonene]

Ready to be reviewed. Though, there seems to be fundamental fixes for 3.14?

[neonene]

FTR, I'm aware of:

• datetime: Stop Exposing Process-Global Objects in the datetime C-API #122184

which is still opaque to me, though.

[neonene]

Updated another PR (#132599), where _datetime no longer has heap types. I think my investigation is finished and any other solution is fine.