[subinterpreters] Don't share Python objects between interpreters

[vstinner]

BPO	40533
Nosy	@vstinner, @ericsnowcurrently, @shihai1991
PRs	[WIP] bpo-40533: Make PyObject.ob_refcnt atomic in subinterpreters #19958 bpo-40533: Disable GC in subinterpreters #19961

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-05-06.15:11:35.602>
labels = ['expert-subinterpreters', '3.9']
title = "[subinterpreters] Don't share Python objects between interpreters"
updated_at = <Date 2022-01-13.17:32:56.034>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2022-01-13.17:32:56.034>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Subinterpreters']
creation = <Date 2020-05-06.15:11:35.602>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 40533
keywords = ['patch']
message_count = 8.0
messages = ['368262', '368274', '368276', '368279', '380105', '388966', '408606', '410499']
nosy_count = 3.0
nosy_names = ['vstinner', 'eric.snow', 'shihai1991']
pr_nums = ['19958', '19961']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue40533'
versions = ['Python 3.9']

[vstinner]

To get one GIL per interpreter (bpo-40512), either PyObject.ob_refcnt member must become an atomic variable, or subinterpreters must not share any object.

Right now, subinterpreters share Python objects. For example, PyModule_Type is declared statically and so shared by all interpreters and so PyModule_Type.tp_mro tuple is accessed in parallel by multiple interpreters. If PyObject.ob_refcnt is not atomic, Py_INCREF() and Py_DECREF() are unsafe and tp_mro tuple can be destroyed whereas it is still used.

I propose to make PyObject.ob_refcnt atomic for now, when Python is built with EXPERIMENTAL_ISOLATED_SUBINTERPRETERS macro defined. It's a temporary workaround until subinterpreters stop sharing objects.

[ericsnowcurrently]

Yep, before per-interpreter GIL is official we must get to the point where *no* PyObject objects are shared.

Making PyObject.ob_refcnt atomic until then (only as part of the experiment) should be fine.

[vstinner]

Yep, before per-interpreter GIL is official we must get to the point where *no* PyObject objects are shared.

I would like to add: "no PyObject objects are shared in the stdlib" ;-)

[vstinner]

New changeset d8135e9 by Victor Stinner in branch 'master':
bpo-40533: Disable GC in subinterpreters (GH-19961)
d8135e9

[vstinner]

For example, PyModule_Type is declared statically and so shared by all interpreters and so PyModule_Type.tp_mro tuple is accessed in parallel by multiple interpreters.

Another example of this issue are parallel Py_INCREF/Py_DECREF calls on the tp_bases member of a type.