bpo-40077: Convert _elementtree types to heap types

[erlend-aasland]

https://bugs.python.org/issue40077

[erlend-aasland]

@corona10 would you mind taking a look at this, when I've sorted out the CI failures?

[tiran]

Tests are failing because subinterpreter re-initialized etree types. You either have to add an initialized guard to type initialization or move the types to module state completely.

[erlend-aasland]

Tests are failing because subinterpreter re-initialized etree types. You either have to add an initialized guard to type initialization or move the types to module state completely.

Ah, thanks, @tiran! I modified the CREATE_TYPE macro with a simple guard. Moving to module state (and multi-phase init) will also be a large PR, so for the reviewers convenience, I think it's best to keep those changes separate.

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[erlend-aasland]

Ref. leaks are now fixed (commit 9210a36):

$ ./python.exe -m test -R 3:5 test_xml_etree test_xml_etree_c ([bpo-40077](https://bugs.python.org/issue40077)/elementtree)cpython.git 0:00:00 load avg: 1.25 Run tests sequentially 0:00:00 load avg: 1.25 [1/2] test_xml_etree beginning 8 repetitions 12345678 ........ 0:00:03 load avg: 1.39 [2/2] test_xml_etree_c beginning 8 repetitions 12345678 ........ == Tests result: SUCCESS == All 2 tests OK. Total duration: 12.4 sec Tests result: SUCCESS 

Rebased onto master & force-pushed; hope that's ok. Ready for review, @tiran.

[erlend-aasland]

FYI, all of these types follow the GC protocol, as described in bpo-42972.

(cc. @vstinner@shihai1991)

[vstinner]

I dislike the temporary situation where we create new heap types but don't clear them at exit.

Would it be possible to write a first PR to convert the _elementtree extension to multiphase init? And maybe also prepare the code for having per module types (having a module state)? (same PR or a different PR) For example, create a state which contains pointers to the static types, and ensure that static types are no longer referenced directly.

[erlend-aasland]

I dislike the temporary situation where we create new heap types but don't clear them at exit.

I agree.

Would it be possible to write a first PR to convert the _elementtree extension to multiphase init? And maybe also prepare the code for having per module types (having a module state)? (same PR or a different PR) For example, create a state which contains pointers to the static types, and ensure that static types are no longer referenced directly.

No problem. I'll create a new PR for multiphase init (bpo 1635741). If it's not too large a change, I'll include module state as well.

[erlend-aasland]

Would it be possible to write a first PR to convert the _elementtree extension to multiphase init?

I did some fiddling with this today, and the problem is that we've already got a (partial) module state, and it often uses PyState_FindModule (via the ET_STATE_GLOBAL macro) to look up the module, in order to fetch the state. Since we can't use PyState_FindModule with multi-phase init, there are two solutions, as I see it:

1. Fetch the state using _PyType_GetModuleByDef and PyType_GetModule, but that requires converting the static types to heap types => the diff explodes => difficult to review
2. (Temporary) convert the existing module state to a global static state, and modify ET_STATE_GLOBAL and get_elementtree_state to return a pointer to the static state. Pro: small PR, easy to review. Con: A small step backwards, module state wise.

What do you think?

[erlend-aasland]

FYI, PEP-687 was just accepted.