gh-124694: Add concurrent.futures.InterpreterPoolExecutor

[ericsnowcurrently]

This is an implementation of InterpreterPoolExecutor that builds on ThreadPoolExecutor.

This assumes that we're okay adding the executor separately from PEP 734. That PEP is about adding a new stdlib module, which is a separate matter from adding the new executor.

Possible future improvements:

• support passing (most) arbitrary functions without pickling
• support passing closures
• optionally exec functions against __main__ instead of the their original module

CC @brianquinlan

• Issue: Add concurrent.futures.InterpreterPoolExecutor #124694

[ericsnowcurrently]

@ZeroIntensity, I've fixed all those Docs things.

[ZeroIntensity]

LGTM. A small nitpick is that it might be a good idea to mention textwrap.dedent in the docs for initializer -- I'm worried that users might run into pesky indentation problems when passing scripts, and textwrap.dedent is a nice way to deal with that.

[ericsnowcurrently]

Good point. It would make sense to automatically call it for users.

[ericsnowcurrently]

@brianquinlan, what are your thoughts on this?

[brianquinlan]

@brianquinlan, what are your thoughts on this?

@ericsnowcurrently I haven't been active for yours but this seems like an excellent addition.

[ericsnowcurrently]

This may have introduced a refleak crash on one of the refleak buildbots: https://buildbot.python.org/#/builders/259/builds/1528. I'm taking a look.

[ZeroIntensity]

Yeah, I noticed the tests for InterpreterPoolExecutor segfaulting on someone elses PR yesterday on something seemingly unrelated. Since this is pure-Python, _interpreters might be buggier than we think :(

[ericsnowcurrently]

In the crash I linked to, it happened when calling _interpqueues.create(), so not _interpreters.

[ericsnowcurrently]

FWIW, the only other failure I've seen with this on refleak buildbots is AMD64 FreeBSD Refleaks 3.x, with 3 successful runs after one failure, which looks like it hung.

The other one I mentioned was on AMD64 RHEL8 Refleaks 3.x, and has had multiple successful runs to go with that one crash.

I don't see any other failures for this on other stable buildbots.

[ericsnowcurrently]

I also haven't been able to reproduce any crash or hang locally yet.

[ZeroIntensity]

I guess it's possible that this only affects AMD?

[ericsnowcurrently]

It's probably a race due to some load profile that has only shown up there.

[ZeroIntensity]

I'll run the tests under valgrind to see if it picks anything up. That will take a while, though. Do you want to revert this in the meantime?

[ZeroIntensity]

Ah, @ericsnowcurrently, I think I found the problem. In most of the _interpqueues methods, something like this is passed to PyArg* as a converter:

qidarg_converter_data qidarg; 

As far as I can tell, default struct initialization is not C standard, so it's values are just junk stack memory. I'm guessing that x86 has some sort of detail that sets stack-allocated structs to NULL, but not on ARM--that's why it's only failing there.

Edit: Oh wait, it's AMD, not ARM. I guess it's a chip-specific issue then, but the point is that it's UB.

[ericsnowcurrently]

Hmm, I'll take a look.

[ericsnowcurrently]

Yeah, that could very well be it. The "label" field would sometimes be a problem if not initialized. I'll fix that.

[ericsnowcurrently]

I've opened gh-125667.

[ZeroIntensity]

Great, I'll review your PR whenever you get to it.

[ericsnowcurrently]

gh-125668

[ericsnowcurrently]

I also noticed a failure on the Android buildbot: https://buildbot.python.org/#/builders/1594/builds/338. I'm looking into it.

[ericsnowcurrently]

#125708

[ericsnowcurrently]

Looks like there's more to do: https://buildbot.python.org/#/builders/1610/builds/198 (AMD64 CentOS9 NoGIL Refleaks). I'm looking into it.

[ericsnowcurrently]

I've opened a new issue to deal with this, so we don't keep going on here: #125716.