[subinterpreters] Multiple interpreters and readline module hook functions.

[grahamd]

BPO	4202
Nosy	@terryjreedy, @vstinner, @ericsnowcurrently
Files	readline.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2008-10-25.02:52:43.004>
labels = ['expert-subinterpreters', 'type-bug']
title = '[subinterpreters] Multiple interpreters and readline module hook functions.'
updated_at = <Date 2020-05-15.01:28:36.744>
user = 'https://bugs.python.org/grahamd'

bugs.python.org fields:

activity = <Date 2020-05-15.01:28:36.744>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Subinterpreters']
creation = <Date 2008-10-25.02:52:43.004>
creator = 'grahamd'
dependencies = []
files = ['27727']
hgrepos = []
issue_num = 4202
keywords = ['patch']
message_count = 5.0
messages = ['75201', '107981', '173809', '173839', '368905']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'vstinner', 'grahamd', 'Winston451', 'eric.snow']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4202'
versions = ['Python 3.5', 'Python 3.6']

[grahamd]

Because the readline module uses PyGILState_Ensure() to facilitate triggering
callbacks into Python code, this would make the ability to use the hook
functions incompatible with use in sub interpreters.

If this is the case, then that readline module cannot be used in sub
interpreters should be documented if not already.

Better still, attempts to register hooks from sub interpreters should result in
an exception. Further, when in use, in sub interpreter, callback hooks should
also not be called if defined, because if defined they would be the hooks from
the main interpreter since variables holding the hooks are static C variables
and shared across all interpreters.

This issue derived from reading of code only and not tested in real program.

[terryjreedy]

The be an issue, tests with currents interpreters are needed. You may be right, but it may be that no one ever uses readline with subinterpreters.

[Winston451]

Hi,

I'm currently developping an application which runs sub interpreters in threads (one sub interpreter per thread). When I try to use readline completion in these sub interpreters it fails because it's not possible to import the "keyword" module (PyEval_GetRestricted() returns true). I think it is due to the fact that readline's on_completion() use PyGILState_Ensure(). It should be possible to use PyEval_RestoreThread with _PyOS_ReadlineTState instead.

[Winston451]

I attached a patch that solve the problem

[vstinner]

See also bpo-15751: [subinterpreters] Make the PyGILState API compatible with subinterpreters.

[ericsnowcurrently]

In the end, we kept the readline module single-phase init, which means it cannot be used in "isolated" subinterpreters (own GIL).
Furthermore, @tonybaloney landed gh-112313 to resolve some of the complexity of the callbacks. I landed gh-121060 (3.13+) to force extensions to always be loaded in the main interpreter first, which resolves some of the other possible complexities. Overall, I believe this issue is sorted out.

If we later want to work toward making the readline module work in multiple interpreters then I recommend we open a new issue.