gh-138342: Use a common utility for visiting an object's type

[ZeroIntensity]

Add _PyObject_VisitType in place of tp_traverse functions that only visit the object's type. This will likely help us catch some bugs through assertions, as we can ensure that this only applies to mutable heap types (and thus are subject to reference cycles). For now, I've kept the assertions relatively relaxed to make it easier to review, but I'll strengthen those assertions and address any bugs in a follow-up.

Sorry about the flood of pings, codeowners.

• Issue: Avoid boilerplate for traversing a mutable heap type #138342

[picnixz]

I think we should be smarter here:

1. Let's first add your utility without it being used.
2. Remove the traverse & GC support for types that are immutable and empty.
3. Add the necessary traverse functions for objects whose class is meant to be mutable.

(1) can be achieved independently but (2) and (3) should perhaps be coordinated depending on what we want to backport. I would suggest that we don't change immutability in 3.13 and 3.14 (so we need the traverse functions for those types) but we can do in 3.15 (in which case we won't need this helper).

[picnixz]

Nevermind, my comment got cross-posted with #116946 (comment). So apparently, it would be safer to not change the immutability of those types for now.

[ZeroIntensity]

Yeah, I found a few other types that don't need this. But as I said in the PR description:

For now, I've kept the assertions relatively relaxed to make it easier to review, but I'll strengthen those assertions and address any bugs in a follow-up.

This PR is pretty big already, and I would like to avoid sprinkling in behavior changes. Once your work with #116946 is done, we can strengthen the assertions in _PyObject_VisitType (e.g., requiring Py_TPFLAGS_IMMUTABLETYPE not to be set).

[picnixz]

Apart from the changes that I will revert in #138338 are removed from this PR, we can merge this. Do you want to review it?

[kumaraditya303]

Most of these are cases where the object actually doesn't needs to have GC at all, they are empty objects with immutable types. Instead of adding this why not just remove GC from them? It will help in performance and reduce gc pressure.

Also in future when improving the assertions, how would you check for empty objects?

[ZeroIntensity]

Also in future when improving the assertions, how would you check for empty objects?

Something like assert(!PyType_HasFeature(tp, Py_TPFLAGS_IMMUTABLETYPE)). We couldn't add it here because of what you stated -- a lot of these don't actually need the GC. Bénédikt is working on finding those types, but I wouldn't go as far as to say that most types here don't need it.

[sergey-miryanov]

Most of these are cases where the object actually doesn't needs to have GC at all, they are empty objects with immutable types. Instead of adding this why not just remove GC from them? It will help in performance and reduce gc pressure.

Also in future when improving the assertions, how would you check for empty objects?

Also, I want to work on this.

[vstinner]

Unless there is a good reason to add this private function to the public C API, I would prefer to move it to the internal C API: see my PR #139734.

If tomorrow, this function becomes popular and very useful outside CPython, I would suggest to make it public. But for now, I would prefer to keep it in the internal C API.

[ZeroIntensity]

I kept it in public headers here to avoid adding #include "pycore_object.h" everywhere, so I have no objections to moving it to the internal headers.

[vstinner]

I kept it in public headers here to avoid adding #include "pycore_object.h" everywhere, so I have no objections to moving it to the internal headers.

Later, it's easy to make it public: add it again to the public C API as PyObject_VisitType(). But if it stays in the public C API as a private name, users will start using it and we cannot change it later :-(