gh-112087: Use QSBR technique for list_new/clear for free-thread build

[corona10]

• Issue: Make list objects thread-safe in --disable-gil builds #112087

[corona10]

➜ cpython git:(gh-115605-qsbr) ✗ ./python.exe -m test test_list test_iter test_sys -R 3:3 Using random seed: 3402256694 Raised RLIMIT_NOFILE: 256 -> 1024 0:00:00 load avg: 2.17 Run 3 tests sequentially 0:00:00 load avg: 2.17 [1/3] test_list beginning 6 repetitions 123456 ...... 0:00:01 load avg: 2.24 [2/3] test_iter beginning 6 repetitions 123456 ...... 0:00:01 load avg: 2.24 [3/3] test_sys beginning 6 repetitions 123456 ...... == Tests result: SUCCESS == All 3 tests OK. Total duration: 6.7 sec Total tests: run=189 skipped=8 Total test files: run=3/3 Result: SUCCESS 

[colesbury]

Thanks for taking this on, and sorry for the delay in reviewing this.

The list_clear change looks good, but I found the other parts difficult to review because the PR includes changes that aren't necessary for using QSBR. In particular:

• The list_good_size is a memory optimization so that requested sizes match mimalloc's internal size classes, but it's not necessary for thread-safety. Additionally, the code assumes that the allocated size is stored in the allocated ob_item array, but that's not implemented yet, so it's hard to tell if the code is correct.
• Similarly, list_ensure_capacity was written to simplify some resize checks given that we will generally want to match mimalloc's size classes rather than do the exact size allocations, but it's not necessary for correctness and not used here as it was used in the nogil-3.12 fork, which makes it harder to review.

I think it'll be easier to review (and maintain) to start with the narrow thread-safety changes and tackle the memory optimizations later.

[colesbury]

We will want the fast path to do this without locking soon.

[colesbury]

What is the motivation behind the refactoring of list_new_prealloc and PyList_New()? It doesn't appear to be necessary for thread-safety. I'm a bit concerned that the change to list_new_prealloc to zero initialize ob_item via PyMem_Calloc may have a performance impact on the default build.

[corona10]

What is the motivation behind the refactoring of list_new_prealloc and PyList_New()? I

Ah, it was the legacy of my first patch, including mimalloc-based optimization.
I revert them all :)

[bedevere-bot]

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 MacOS M1 Refleaks NoGIL 3.x has failed when building commit fb5e034.

What do you need to do:

1. Don't panic.
2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1368/builds/377) and take a look at the build logs.
4. Check if the failure is related to this commit (fb5e034) or if it is a false positive.
5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1368/builds/377

Failed tests:

• test.test_concurrent_futures.test_shutdown
• test_sched

Test leaking resources:

• test_sched: memory blocks
• test_shutdown: memory blocks

Summary of the results of the build (if available):

==

Click to see traceback logs

Note: switching to 'fb5e0344e41788988171f31c6b8d4fd1a13b9041'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c <new-branch-name> Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at fb5e0344e4 gh-112087: Use QSBR technique for list_new/clear for free-thread build (gh-115875) Switched to and reset branch 'main' In file included from ./Modules/tkappinit.c:17: In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99: /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] int (*free_private)(); /* called to free private storage */^ void In file included from ./Modules/_tkinter.c:52: In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99: /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] int (*free_private)(); /* called to free private storage */^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] struct _XImage *(*create_image)();^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] struct _XImage *(*create_image)();^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] XID (*resource_alloc)(); /* allocator function */^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] int (*synchandler)(); /* Synchronization handler */^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] Bool (*event_vec[128])(); /* vector for wire to event */^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h::453497::2325:: warning: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] XID (*resource_alloc)(); /* allocator function */ Status (*wire_vec[128])(); /* vector for event to wire */^^ void void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] Bool (**error_vec)(); /* vector for wire to error *//opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h : ^471 : void20 : warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] int (*synchandler)(); /* Synchronization handler */^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] int (*savedsynchandler)(); /* user synchandler when Xlib usurps */^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] Bool (*event_vec[128])(); /* vector for wire to event */^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] Status (*wire_vec[128])(); /* vector for event to wire */^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] Bool (**error_vec)(); /* vector for wire to error */^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] int (*savedsynchandler)(); /* user synchandler when Xlib usurps */^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] typedef void (*XIMProc)(); ^ void /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes] typedef void (*XIMProc)(); ^ void 9 warnings generated. 9 warnings generated. make: *** [buildbottest] Error 2

[corona10]

• test_sched: Already leaked but increased -> test_sched leaked [1, 0, 0] memory blocks, sum=1 (this is fine) to test_sched leaked [5, 4, 4] memory blocks, sum=13
• test.test_concurrent_futures.test_shutdown looks fine in my loacl.

Using random seed: 2687821224 Raised RLIMIT_NOFILE: 256 -> 1024 0:00:00 load avg: 2.41 Run 1 test sequentially 0:00:00 load avg: 2.41 [1/1] test_concurrent_futures.test_shutdown beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more) 123:456 XX1 ... test_concurrent_futures.test_shutdown passed in 1 min 13 sec == Tests result: SUCCESS == 1 test OK. Total duration: 1 min 13 sec Total tests: run=47 skipped=3 Total test files: run=1/1 Result: SUCCESS 

[corona10]

Root cause: _PyMem_ProcessDelayed is not executed

cpython/Objects/obmalloc.c

Lines 1021 to 1023 in 5dc8c84

	if (buf->wr_idx==WORK_ITEMS_PER_CHUNK){
	_PyMem_ProcessDelayed((PyThreadState*)tstate);
	}

@colesbury Is there a way to trigger _PyMem_ProcessDelayed by force?

[corona10]

With #116237 (Not sure that it will be the proper solution)

➜ cpython git:(gh-112087-leak-tmp) ✗ ./python.exe -m test test_sched -R 3:3 Using random seed: 2623432415 Raised RLIMIT_NOFILE: 256 -> 1024 0:00:00 load avg: 2.50 Run 1 test sequentially 0:00:00 load avg: 2.50 [1/1] test_sched beginning 6 repetitions. Showing number of leaks (. for 0 or less, X for 10 or more) 123:456 XX4 5.4 test_sched leaked [0, 0, 1] references, sum=1 (this is fine) test_sched leaked [5, -16, 4] memory blocks, sum=-7 (this is fine) == Tests result: SUCCESS == 1 test OK. Total duration: 3.4 sec Total tests: run=11 Total test files: run=1/1 Result: SUCCESS 

[corona10]

Okay #116237 fix the issue, but I prefer #116238