GH-93533: Shrink the LOAD_ATTR caches

[brandtbucher]

This adds a fixed array of cached methods to the PyTypeObject struct. This approach saves memory if there are ~23x more LOAD_ATTR sites than there are types.

A size of 16 was chosen because:

• A size of 8 is 1% slower and has a 81.0% LOAD_ATTR hit rate on pyperformance, with 32.5% of specialization failures due to a too-small cache.
• A size of 16 is 0% faster and has a 82.2% LOAD_ATTR hit rate on pyperformance, with 8.7% of specialization failures due to a too-small cache.
• A size of 32 is 0% faster and has a 82.6% LOAD_ATTR hit rate on pyperformance, with 3.3% of specialization failures due to a too-small cache.

A flexible buffer was also considered, but that was 1% slower, likely due to the additional indirection and management of the buffer.

• Issue: The inline cache for LOAD_ATTR is too large. #93533

[TeamSpen210]

So if I'm understanding this correctly, for each class only 16 methods/attributes can be specialised, with everything after just failing? If so 16 seems fairly small, especially for larger libraries/applications. Something like ndarray or flask.Flask for instance has 50+ methods. I could imagine a larger application using different sets of methods in different areas, filling up the cache. That wouldn't be the sort of thing showing up on benchmarks.

[brandtbucher]

So if I'm understanding this correctly, for each class only 16 methods/attributes can be specialised, with everything after just failing?

Correct (not including instance attributes). I definitely prefer a growable cache to this approach, except that the numbers we have are slower.

I'm also open to using a larger number of entries, like 32 or 64, if others feel similarly.

[brandtbucher]

I'm also not sure if this causes problems for interpreter isolation. Maybe putting this sort of state on static built-in types is a bad idea, whether it's resizable or not.

CC @ericsnowcurrently

[hugovk]

(Docs now passing now #103019 is merged, and updating this with main. Thanks for flagging!)

[markshannon]

From a correctness point of view, adding the cache to static classes should be fine. All the attributes of superclasses of static classes must also be static.
If they aren't that's a bug, and this isn't the place to fix it.
No harm in adding some asserts, though.

However, we would like static classes to be const so that they can be properly shared, which would mean that the cache would need to be pre-populated with all attributes of the class and all its superclasses.
The caches would be a bit big, but not that bad: len(object.__dict__ | int.__dict__) == 74; much smaller if we exclude special methods.

[brandtbucher]

Closing because of the various issues outlined above.