[subinterpreters] Make the PyGILState API compatible with subinterpreters

[ncoghlan]

BPO	15751
Nosy	@mhammond, @jcea, @ronaldoussoren, @ncoghlan, @pitrou, @vstinner, @phsilva, @asvetlov, @ericsnowcurrently, @soltysh, @seberg, @erlend-aasland, @Felk, @h-vetinari

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-08-21.05:18:00.735>
labels = ['expert-subinterpreters', 'expert-C-API', 'type-feature', '3.10']
title = '[subinterpreters] Make the PyGILState API compatible with subinterpreters'
updated_at = <Date 2022-01-07.18:32:22.492>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2022-01-07.18:32:22.492>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['C API', 'Subinterpreters']
creation = <Date 2012-08-21.05:18:00.735>
creator = 'ncoghlan'
dependencies = []
files = []
hgrepos = []
issue_num = 15751
keywords = []
message_count = 49.0
messages = ['168742', '168759', '168764', '168766', '168768', '168769', '168771', '168772', '168776', '168779', '168780', '168787', '168789', '168790', '168792', '168812', '168815', '168816', '168818', '168832', '169008', '169010', '169013', '169014', '169278', '169280', '169284', '169286', '169310', '169329', '169330', '169332', '169333', '169334', '169352', '169354', '169359', '198115', '206016', '206017', '206042', '206048', '261779', '368885', '368904', '368906', '375134', '380325', '409995']
nosy_count = 17.0
nosy_names = ['mhammond', 'jcea', 'ronaldoussoren', 'ncoghlan', 'pitrou', 'vstinner', 'phsilva', 'grahamd', 'asvetlov', 'python-dev', 'eric.snow', 'maciej.szulik', 'seberg', 'erlendaasland', 'felk', 'h-vetinari', 'gabrielhae']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue15751'
versions = ['Python 3.10']

Linked PRs

• gh-59956: Clarify GILState-related Code #101161
• gh-59956: Fix Function Groupings in pystate.c #101172
• gh-59956: Allow the "Trashcan" Mechanism to Work Without a Thread State #101209
• gh-59956: Clarify Runtime State Status Expectations #101308
• gh-59956: Partial Fix for GILState API Compatibility with Subinterpreters #101431
• gh-59956: Add a Test to Verify GILState Matches the "Current" Thread State #101625

[ncoghlan]

Currently, modules that use the PyGILState* APIs cannot be used with mod_wsgi, as mod_wsgi uses the subinterpreter support.

Graham Dumpleton and I spent some time discussing this at PyCon AU 2012, and we believe that the incompatibility can be resolved with a single API in the core: a function that mod_wsgi can call to set the interpreter used by the GIL state APIs to implicitly create new thread states.

This is still only a seed of an idea (and it's entirely possible we've missed something), but it's a place to start in resolving this longstanding incompatibility.

[grahamd]

Just to clarify. One can still tell WSGI applications under mod_wsgi to run in the main interpreter and in that case modules using PyGILState* do work. By default though, sub interpreters are used as noted.

The mechanism for forcing use of main interpreter is the directive:

WSGIApplicationGroup %{GLOBAL}

Some details about this issue can be found in:

http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API

[pitrou]

a function that mod_wsgi can call to set the interpreter used by the
GIL state APIs to implicitly create new thread states.

How would it work?

[ncoghlan]

It would twiddle the autoInterpreterState and autoTLSkey entries in the pystate.c global variables to point to a different subinterpreter.

As I understand the situation, mod_wsgi doesn't need arbitrary externally created threads to be able to call back into arbitrary subinterpreters, it just needs to be able to direct externally created threads in a process to a subinterpreter other than the main one.

Graham, looking at the current impl - have you experimented with just calling _PyGILState_Init() with the interpreter state and current thread state for the desired subinterpreter to see what happens?

I think the new method could just be a cleaner combination of _PyGILState_ReInit and _PyGILState_Init. If I'm right, then calling _PyGILState_Init should convert the current crashes and deadlocks into a relatively less harmful memory leak (since the old entry in the TLS won't get deleted properly).

[ncoghlan]

Graham, even better would be if you could try the following combination:

_PyGILState_Fini();
_PyGILState_Init(si, st);

(where si and st are the interpreter state and thread state for the target subinterpreter)

If a new PyGILState_SwitchInterpreter API is going to be able to solve this in 3.4, then I believe those private APIs should be enough to make it possible in *current* versions.

If those private APIs *aren't* enough, then I'm missing something and this isn't going to be as easy as I thought.

[pitrou]

As I understand the situation, mod_wsgi doesn't need arbitrary
externally created threads to be able to call back into arbitrary
subinterpreters, it just needs to be able to direct externally created
threads in a process to a subinterpreter other than the main one.

But how does it know that the right externally created thread will point
to the right subinterpreter? The OS is free to schedule threads as it
desires.

[ncoghlan]

Just as they do today, all externally created threads will still go to *one* interpreter when they hit PyGILState_Ensure(). It's just that interpreter won't be the main one.

Since the whole point of the PyGILState API is to support threads that don't have a previously created thread state, there's no getting around the requirement to have a single blessed interpreter that handles all externally created threads in a given process.

It will be up to mod_wsgi (and any other embedding application that uses the new function) to make sure it calls this at a time when there aren't any existing calls to PyGILState that would be disrupted. (Assuming we can't figure out a locking scheme that *ensures* no such threads are running when the switch occurs)

[pitrou]

Just as they do today, all externally created threads will still go to
*one* interpreter when they hit PyGILState_Ensure(). It's just that
interpreter won't be the main one.

Uh but how does it solve the issue? (unless you create a mod_wsgi app
with only a single worker thread)

[ncoghlan]

My understanding of the mod_wsgi architecture is that it uses subinterpreters to maintain a persistent process, while still providing a relatively pristine interpreter state to handle each new request. This means even when you're using multiple processes with a single request handling thread per process, you're running a subinterpreter unless you explicitly configure mod_wsgi to always run in the main interpreter (which, I believe, will result in additional state persistence across requests).

The proposed API change can only fix scenarios where *at a given point in time*, *all* PyGILState_Ensure calls should be directed to a particular subinterpreter. The target subinterpreter may change *later*, but there still cannot be two desired targets within the same process at the same time.

However, at the moment, PyGILState doesn't even allow that - all externally created threads are handled in the *main* interpreter even if that isn't what the embedding application wants.

[ncoghlan]

Sorry, I mischaracterised the way mod_wsgi works slightly. However, my understanding is still that the scope of this particular fix is merely to allow all external threads to be redirected to a different subinterpreter at various times over the life of a process. It does not need to allow different external threads to be redirected to different subinterpreters.

(Note: I am assuming that any hooks Apache/mod_wsgi has into external thread creation could already just create an appropriate thread state if that was the desired behaviour. It may be I'm incorrect on this, and what Graham really wants is the ability to change the target interpreter state just for the current thread. However, if that's what he wants, then there's additional background info I need on mod_wsgi and its ability to influence thread creation within a process, because I didn't get the impression on the weekend that that is what he was after)

[pitrou]

My understanding of the mod_wsgi architecture is that it uses
subinterpreters to maintain a persistent process, while still
providing a relatively pristine interpreter state to handle each new
request.

I don't think that's true. On hg.python.org, the hglookup application
keeps a cached mapping of changeset ids to repo URLs, which wouldn't be
possible if its interpreter was re-bootstrapped for each new request.

[ncoghlan]

s/slightly/completely/ (I believe my description of the core problem was right, but my description of why that problem exists was wrong - it actually has to do with the way mod_wsgi handles virtual hosts and endpoints)

If we expose an official way to switch the destination of the PyGILState APIs, then what it means is that mod_wsgi can, over the lifecycle of a single process in the pool, *switch* the virtual host and WSGI endpoint that process is handling by changing the active interpreter. There are still some extension modules where that won't work (because they create persistent threads that periodically call back into Python, so they may still end up calling back in to the wrong interpreter), but it will allow those that just do callbacks from external worker threads (or other operations that are similarly bound by the lifecycle of a request) to start working properly.

[pitrou]

If we expose an official way to switch the destination of the
PyGILState APIs, then what it means is that mod_wsgi can, over the
lifecycle of a single process in the pool, *switch* the virtual host
and WSGI endpoint that process is handling by changing the active
interpreter.

I don't understand what that means. It's the OS which switches threads,
including interpreter threads.

(by the way, the real fix to the GILState vs. subinterpreters issue
would be to create new API functions which take an interpreter as
argument, so that you have a distinct TLS key per interpreter)

[ncoghlan]

Umm, no. The whole point of the GILState API is that you can call it from a thread which knows *nothing* about Python.

It will then use the *process global* state in the pystate.c statics to initialise that thread with a Python thread state. Currently, that thread state will always be for the main Python interpreter for the process.

All Graham wants is an officially supported way to change which interpreter the pystate.c globals reference *without* shutting down and reinitialising Python completely.

There are going to be limitations on how effective this will be - it still won't support *all* extension modules that use the PyGILState APIs. It will, however, support many more of them than the status quo (which is zero, unless you force your WSGI app to use the main interpreter, which has its own problems).

And you absolutely can control when the OS switches threads - controlling that is what synchronisation primitives are *for*.