gh-95991: Add some infrastructure for testing Limited API in _testcapi

[encukou]

• Limited API needs to be enabled per source file
• Some builds don't support Limited API, so Limited API tests must be skipped on those builds
  (currently this is Py_TRACE_REFS, but that may change.)
• Py_LIMITED_API must be defined before <Python.h> is included.

This puts the hoop-jumping in testcapi/parts.h, so individual
test files can be relatively simple. (Currently that's only
vectorcall_limited.c, imagine more.)

• Issue: Test Limited API in _testcapi module #95991

[encukou]

I'm not sure about best practices for test/support. @vstinner or @erlend-aasland, any comments on the @requires_limited_api decorator?

[encukou]

I also plan to add some instructions for the devguide, mentioning the structure for a Modules/testcapi/*.c file:

#definePy_LIMITED_API 0x03... #include"parts.h"#ifdefLIMITED_API_AVAILABLE (allcontentgoeshere) #endif

and @requires_limited_api for any Python parts.

[erlend-aasland]

Looks good! You might want to document the test decorator in Doc/library/test.rst.

[encukou]

This happens to move #undef NDEBUG and so it should fix assert warnings reported in #96017

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @encukou for commit b3b50c0 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[encukou]

I don't understand the failure on the Windows buildbot:

 6>PrepareForBuild: Creating directory "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\_testcapi.tlog\". InitializeBuildStatus: Creating "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\_testcapi.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified. ClCompile: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\HostX86\arm64\CL.exe /c /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\Include" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\Include\internal" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PC" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\\" /Zi /nologo /W3 /WX- /diagnostics:column /MP5 /O2 /Oi /Oy- /GL /D WIN32 /D "PY3_DLLNAME=L\"python3\"" /D _WIN32 /D NDEBUG /D _ARM64_WINAPI_PARTITION_DESKTOP_SDK_AVAILABLE=1 /D _WINDLL /GF /Gm- /MD /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\\" /Fd"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\vc143.pdb" /external:W3 /Gd /TC /analyze- /FC /errorReport:queue /utf-8 ..\Modules\_testcapimodule.c ..\Modules\_testcapi\vectorcall.c ..\Modules\_testcapi\vectorcall_limited.c ..\Modules\_testcapi\heaptype.c ..\Modules\_testcapi\unicode.c _testcapimodule.c vectorcall.c vectorcall_limited.c heaptype.c unicode.c ResourceCompile: C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x86\rc.exe /D "ORIGINAL_FILENAME=\\\"_testcapi.pyd\\\"" /D FIELD3=100 /D NDEBUG /l"0x0409" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PC" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\Include" /I"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\\" /nologo /fo"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\python_nt.res" ..\PC\python_nt.rc Link: C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.31.31103\bin\HostX86\arm64\link.exe /ERRORREPORT:QUEUE /OUT:"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\arm64\_testcapi.pyd" /INCREMENTAL:NO /NOLOGO /LIBPATH:"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\arm64\\" advapi32.lib shell32.lib ole32.lib oleaut32.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /NODEFAULTLIB:LIBC /MANIFEST:NO /DEBUG /PDB:"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\arm64\_testcapi.pdb" /SUBSYSTEM:WINDOWS /LTCG /LTCGOUT:"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\_testcapi.iobj" /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\arm64\_testcapi.lib" /MACHINE:ARM64 /OPT:REF,NOICF /DLL "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\python_nt.res" "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\_testcapimodule.obj" "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\vectorcall.obj" "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\vectorcall_limited.obj" "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\heaptype.obj" "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\obj\312arm64_Release\_testcapi\unicode.obj" "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\arm64\python312.lib" 6>LINK : fatal error LNK1104: cannot open file 'python3.lib' [C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\_testcapi.vcxproj] 6>Done Building Project "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\_testcapi.vcxproj" (Build target(s)) -- FAILED. 1>Done Building Project "C:\Workspace\buildarea\pull_request.linaro-win-arm64.nondebug\build\PCbuild\pcbuild.proj" (Build target(s)) -- FAILED. 

Windows failures need to be resolved

[encukou]

@python/windows-team, do you know what's wrong here, by any chance?

[zooba]

It's probably getting built before the limited API DLL (python3.dll). We need to force a dependency into the project file to ensure they are built in the right order.

Add the below to _testcapi.vcxproj underneath the existing <ProjectReference Include="pythoncore.vcxproj"> that's in there:

 <ProjectReference Include="python3dll.vcxproj"> <Project>{885d4898-d08d-4091-9c40-c700cfe3fc5a}</Project> <ReferenceOutputAssembly>false</ReferenceOutputAssembly> </ProjectReference> 

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @encukou for commit ee59cb3 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[encukou]

Woohoo! That worked! Thanks for the wincantation, Steve, and thanks Erlend for the review!

The other buildbot failures are unrelated, and being handled in #96006 and #95898

[vstinner]

When Py_TRACE_REFS is defined, it was discussed once to move the double linked list of objects out of PyObject, to keep the ABI compatiiblity. It shouldn't be hard to store it somewhere else. But so far, nobody proposed a concrete implementation of this idea.