bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module.#1748

ericsnowcurrently · 2017-05-23T04:09:21Z

https://bugs.python.org/issue32604

pitrou · 2017-05-25T20:05:04Z

Doc/library/_interpreters.rst

@@ -0,0 +1,69 @@
+:mod:`_interpreters` --- Low-level interpreters API


There should be a warning about the module being private, and/or the API being subject to change without notice, IMHO.

Good point. For the most part I was following the precedent of the _thread module. However, in this case the module is relatively unstable still and there is no "interpreters" module yet. So a warning makes sense.

Given the not-yet-approved PEP, I think it would be a good idea to use a name that makes it clear that this really is intended as an API for the CPython test suite, rather than for general use.
That would mean going with something like _xxsubinterpreters for now.

Note: one consequence of that is that redistributors that remove the test suite would likely remove this module as well, and I think that's a desirable outcome: we want this as a learning tool for the further development of PEP 554 as a proposal for 3.8 (and to find and fix bugs in the subinterpreter support in 3.7), not as an inadvertent commitment to a particular API.

pitrou · 2017-05-25T20:06:50Z

Lib/test/test__interpreters.py

+import threading
+import _interpreters
+def f():
+ _interpreters.run_string(id, dedent('''


I don't understand where the id variable is coming from... is it the built-in id function?

Good catch. That's a typo that I didn't notice because the resulting exception in the thread does not cause the process to fail.
FYI, adding the missing "id = _interpreters.create()" line causes the test to fail (correctly); the subinterpreter didn't get cleaned up before we reached Py_Finalize(), resulting in Py_FatalError. I'll sort that out separately.

1st1 · 2018-01-20T05:37:37Z

Doc/library/_interpreters.rst

Can we call the function run as per the PEP?

Note that the PEP is all about the high-level API we want to expose to users. In contrast, run_string() is the low-level function focused strictly as an analog to PyRun_String*(). As we support other runnable things (e.g. code objects, functions), we will add corresponding low-level functions. The eventual high-level method (i.e. Interpreter.run() will call the appropriate low-level function.
Also, I'm removing the docs for now.

1st1 · 2018-01-20T05:43:53Z

Include/internal/pystate.h

The actual layout of the struct should go into an "internal" header file:
"Include/pystate.h":
typedefstruct_xid_PyCrossInterpreterData;
"Include/internal/pystate.h":
struct_xid{void*data; // etc }

I haven't touched Include/pystate.h. If you are suggesting that I move the typedef (but not the layout) there, I don't see any benefit to that for 3.7. This is not a public API for now.

1st1 · 2018-01-20T05:47:46Z

Modules/_interpretersmodule.c

You don't need to check for NULL here, PyThreadState_Get aborts if it can't lookup the state.

1st1 · 2018-01-20T05:48:33Z

Modules/_interpretersmodule.c

per PEP 7 please use curly braces always :)

1st1 · 2018-01-20T05:50:44Z

Modules/_interpretersmodule.c

Check if shared is NULL

1st1 · 2018-01-20T05:53:19Z

Modules/_interpretersmodule.c

Redundant? And need to free err?

I'd meant to return err on the next line rather than NULL. :)

ericsnowcurrently · 2018-01-27T19:46:18Z

FYI, I've temporarily removed the Windows support to see if anything else breaks. I'll add it back in soon. I do not plan on merging this without it.

ericsnowcurrently · 2018-01-27T23:22:19Z

@zooba I suspect my problem is with Py_BUILD_CORE. When I don't set it I don't get the symbols out of Include/internal. When I do set it I end up missing stuff from python37.lib at link time. I tried adding a "Link" directive right below where I set Py_BUILD_CORE, but that did not help. :/ Any ideas on what I need to do?

ericsnowcurrently · 2018-01-29T16:17:48Z

@python/windows-team Any of you have ideas on how I can get this PR to link on Windows?

ericsnowcurrently · 2018-01-30T00:58:03Z

In the interest of the feature freeze I'm going to land the PR without Windows support. :/ The risk for beta1 is low since the PR does not have any OS-specific code and the new module is only for use in the test suite. I'll work on getting Windows support done ASAP.

ncoghlan · 2018-01-30T03:37:15Z

Include/internal/pystate.h

+//
+// We use the ID rather than the PyInterpreterState to avoid issues
+// with deleted interpreters.
+int64_tinterp;


Something we'll need to keep an eye on here is not re-using interpreter IDs within the life of a process (otherwise we'll still run into issues with deleted interpreters).
Not that I've ever worked on systems where we introduced bugs by adding a freelist for objects where we assumed IDs would never be re-used or anything ;)

We picked int64_t so that it was both really large and would overflow to negative. When the next ID is negative PyInterpreterState_New() raises RuntimError.

the-knights-who-say-ni added the CLA signed label May 23, 2017

ericsnowcurrently force-pushed the low-level-interpreters-module branch from e0611a8 to bee1baaCompare May 23, 2017 04:38

ericsnowcurrently changed the title ~~Expose the subinterpreters C-API in a "private" stdlib module.~~bpo-30439: Expose the subinterpreters C-API in a "private" stdlib module.May 23, 2017

ericsnowcurrently force-pushed the low-level-interpreters-module branch from bee1baa to 060a720Compare May 23, 2017 23:42

ericsnowcurrently mentioned this pull request May 24, 2017
bpo-30439: Add some helpful low-level functions for subinterpreters. #1802
Closed

pitrou reviewed May 25, 2017
View reviewed changes

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 060a720 to 2fd0e64Compare August 22, 2017 20:01

ericsnowcurrently force-pushed the low-level-interpreters-module branch 2 times, most recently from 114bda4 to 52ca360Compare September 7, 2017 17:58

Mariatta added needs rebase and removed needs rebase labels Oct 9, 2017

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 52ca360 to 9991d75Compare November 27, 2017 23:24

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 9991d75 to e9d9b04Compare December 5, 2017 01:29

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 81de3c8 to bfc025fCompare December 12, 2017 05:12

ericsnowcurrently force-pushed the low-level-interpreters-module branch 2 times, most recently from e05d02e to 396e294Compare December 23, 2017 00:30

ericsnowcurrently force-pushed the low-level-interpreters-module branch 3 times, most recently from 5266080 to c86647aCompare January 13, 2018 00:49

ericsnowcurrently force-pushed the low-level-interpreters-module branch 2 times, most recently from ff4ae72 to 0981dbfCompare January 20, 2018 01:44

ericsnowcurrently changed the title ~~bpo-30439: Expose the subinterpreters C-API in a "private" stdlib module.~~bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module.Jan 20, 2018

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 0981dbf to 0be6314Compare January 20, 2018 02:13

1st1 reviewed Jan 20, 2018
View reviewed changes

ericsnowcurrently added 10 commits January 27, 2018 19:12

Add typedefs.
2df5aff

_shared_exception -> _sharedexception
9bb05cf

Free memory in the error case.
633a3d3

Whitelist instead of blacklist for richcompare ops.
74e7241

Decref in the error case.
bb8619a

Add a Misc/NEWS entry.
d785573

Do not try to print a bytes object.
f856035

Drop some outdated comments.
ba94571

Wrap long lines.
9065001

Fix some compiler warnings.
2ebc68c

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 8ae8507 to 2ebc68cCompare January 27, 2018 19:15

Always use PyMem_NEW().
48291af

ericsnowcurrently force-pushed the low-level-interpreters-module branch 2 times, most recently from ebbf591 to 9718b91Compare January 27, 2018 21:07

Do not try to allocate too much space.
39df95e

ericsnowcurrently force-pushed the low-level-interpreters-module branch 2 times, most recently from edec009 to 092fcccCompare January 27, 2018 22:50

Add Windows build support.
b07e928

ericsnowcurrently force-pushed the low-level-interpreters-module branch from 092fccc to b07e928Compare January 27, 2018 23:19

Drop Windows build support (for now).
1bfed0f

ericsnowcurrently merged commit 7f8bfc9 into python:masterJan 30, 2018

ericsnowcurrently mentioned this pull request Jan 30, 2018
bpo-32604: Add the _xxsubinterpreters extension module under Windows. #5436
Closed

ncoghlan reviewed Jan 30, 2018
View reviewed changes

ericsnowcurrently deleted the low-level-interpreters-module branch February 3, 2018 04:54

ericsnowcurrently mentioned this pull request Aug 24, 2018
add low-level module (_xxsubinterpreters) ericsnowcurrently/multi-core-python#1
Closed

ericsnowcurrently mentioned this pull request Apr 10, 2022
[subinterpreters] PEP 554 implementation: add interpreters module #76785
Closed

		@@ -0,0 +1,69 @@
		:mod:`_interpreters` --- Low-level interpreters API

Uh oh!

bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module.#1748

bpo-32604: Expose the subinterpreters C-API in a "private" stdlib module. #1748

Uh oh!

Conversation

ericsnowcurrently commented May 23, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ericsnowcurrently commented Jan 27, 2018

Uh oh!

ericsnowcurrently commented Jan 27, 2018

Uh oh!

ericsnowcurrently commented Jan 29, 2018

Uh oh!

ericsnowcurrently commented Jan 30, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ericsnowcurrently commented May 23, 2017•
edited
Loading