Skip to content

Conversation

@JunyiXie
Copy link
Contributor

@JunyiXieJunyiXie commented Mar 19, 2021

https://bugs.python.org/issue43551

fix PyImport_Import use static silly_list under building Python with --with-experimental-isolated-subinterpreters share silly_list in multi subinterpreters cause crash.

Under the sub interpreters parallel, PyObject_CallFunction clean stack,
Py_DECREF(stack[i]), Py_DECREF silly_list is not thread safe. cause crash

PyObject * PyImport_Import(PyObject *module_name){PyThreadState *tstate = _PyThreadState_GET(); static PyObject *silly_list = NULL; ... /* Initialize constant string objects */ if (silly_list == NULL){import_str = PyUnicode_InternFromString("__import__"); if (import_str == NULL) return NULL; builtins_str = PyUnicode_InternFromString("__builtins__"); if (builtins_str == NULL) return NULL; silly_list = PyList_New(0); if (silly_list == NULL) return NULL} ... /* Call the __import__ function with the proper argument list Always use absolute import here. Calling for side-effect of import. */ r = PyObject_CallFunction(import, "OOOOi", module_name, globals, globals, silly_list, 0, NULL); 

https://bugs.python.org/issue43551

@JunyiXie
Copy link
ContributorAuthor

Hello, this change seems might not need to update news.

Python/import.c Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"silly list" is not a great name, I propose to rename it to "from_list".

Also you must put a DECREF somewhere, otherwise the code will leak.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--with-experimental-isolated-subinterpreters option is supposed to remain secret!

Either remove your NEWS entry, or just say "subinterpreters". For example, "Fix the import function for subinterpreters." I'm not sure that it's worth it to document every single subinterpreter fix.

Python/import.c Outdated
Comment on lines 1749 to 1750
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

static variables are bad, they don't work in subinterpreters. You can use the _Py_IDENTIFIER() API with _PyUnicode_FromId() (return a borrowed reference), it is compatible with subinterpreters.

@JunyiXie
Copy link
ContributorAuthor

JunyiXie commented Mar 20, 2021

@vstinner thanks for your review. I have fix these problems.

Copy link
Member

@vstinnervstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, it's way better, thanks! I have remarks on the coding style.

Python/import.c Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can define the variable here (same for the 2 other variables) and add a comment to mention that it's a borrowed ref

Suggested change
import_str=_PyUnicode_FromId(&PyId___import__);
PyObject*import_str=_PyUnicode_FromId(&PyId___import__);// borrowed ref

Copy link
Member

@vstinnervstinner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a refleak (unrelated to your PR) at:

 /* No globals -- use standard builtins, and fake globals */ builtins = PyImport_ImportModuleLevel("builtins", NULL, NULL, NULL, 0); if (builtins == NULL) return NULL; // <=== HERE 

Please replace return NULL with goto err and add braces (PEP 7).

You may also replace return NULL with goto err at:

 PyObject *from_list = PyList_New(0); if (from_list == NULL){return NULL} 

You might also remove trailing spaces at line 1761 :-D (my text editor highlights them ;-))

under building Python with --with-experimental-isolated-subinterpreters share silly_list in multi subinterpreters cause crash.
@vstinnervstinner merged commit 88d9983 into python:masterMar 22, 2021
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 macOS 3.x has failed when building commit 88d9983.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/366/builds/969) and take a look at the build logs.
  4. Check if the failure is related to this commit (88d9983) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/366/builds/969

Failed tests:

  • test_importlib

Failed subtests:

  • test_multiprocessing_pool_circular_import - test.test_importlib.test_threaded_import.ThreadedImportTests

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

408 tests OK.

10 slowest tests:

  • test_concurrent_futures: 4 min 15 sec
  • test_unparse: 3 min 1 sec
  • test_multiprocessing_spawn: 2 min 48 sec
  • test_tokenize: 2 min 21 sec
  • test_multiprocessing_forkserver: 2 min 3 sec
  • test_lib2to3: 1 min 59 sec
  • test_asyncio: 1 min 42 sec
  • test_unicodedata: 1 min 36 sec
  • test_capi: 1 min 34 sec
  • test_io: 1 min 10 sec

1 test failed:
test_importlib

18 tests skipped:
test_devpoll test_epoll test_gdb test_ioctl test_msilib
test_multiprocessing_fork test_ossaudiodev test_smtpnet test_spwd
test_ssl test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_importlib

Total duration: 24 min 45 sec

Click to see traceback logs
Traceback (most recent call last): File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import script_helper.assert_python_ok(fn) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 160, in assert_python_okreturn _assert_python(True, *args, **env_vars) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 145, in _assert_python res.fail(cmd_line) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 72, in failraiseAssertionError("Process return code is %d\n"AssertionError: Process return code is 1 command line: ['/Users/buildbot/buildarea/3.x.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py'] Traceback (most recent call last): File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in twith multiprocessing.Pool(1): File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Poolreturn Pool(processes, initializer, initargs, maxtasksperchild, Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in twith multiprocessing.Pool(1): File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in twith multiprocessing.Pool(1): File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t Traceback (most recent call last): File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 212, in __init__self._repopulate_pool() File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Poolreturn Pool(processes, initializer, initargs, maxtasksperchild, File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Poolreturn Pool(processes, initializer, initargs, maxtasksperchild, File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Poolreturn Pool(processes, initializer, initargs, maxtasksperchild, File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 212, in __init__self._repopulate_pool() File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 196, in __init__self._change_notifier =self._ctx.SimpleQueue() File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in twith multiprocessing.Pool(1): File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_poolreturnself._repopulate_pool_static(self._ctx, self.Process, File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 212, in __init__self._repopulate_pool() File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_poolreturnself._repopulate_pool_static(self._ctx, self.Process, File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 113, in SimpleQueuereturn SimpleQueue(ctx=self.get_context()) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_poolreturnself._repopulate_pool_static(self._ctx, self.Process, File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static w.start() File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Poolreturn Pool(processes, initializer, initargs, maxtasksperchild, File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static w.start() File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/queues.py", line 347, in __init__self._wlock = ctx.Lock() File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 212, in __init__self._repopulate_pool() File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/process.py", line 121, in startself._popen =self._Popen(self) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static w.start() File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/process.py", line 121, in startself._popen =self._Popen(self) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 68, in Lockreturn Lock(ctx=self.get_context()) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/process.py", line 121, in startself._popen =self._Popen(self) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 284, in _Popenreturn Popen(process_obj) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_poolreturnself._repopulate_pool_static(self._ctx, self.Process, File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 284, in _Popenreturn Popen(process_obj) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/synchronize.py", line 162, in __init__ SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 284, in _Popenreturn Popen(process_obj) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__super().__init__(process_obj) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/synchronize.py", line 57, in __init__ sl =self._semlock = _multiprocessing.SemLock( File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__super().__init__(process_obj) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__self._launch(process_obj) OSError: [Errno 24] Too many open files /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 29 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' --- Traceback (most recent call last): File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 196, in __init__ File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 113, in SimpleQueue File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/queues.py", line 341, in __init__self._reader, self._writer = connection.Pipe(duplex=False) File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/connection.py", line 532, in Pipe fd1, fd2 = os.pipe() OSError: [Errno 24] Too many open files /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 110 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-56i6i4ap': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-0au5otkl': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vcv0xwbi': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vxfb4ks9': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-5e2_0z1f': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-6vsgax4k': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-mq51g4b_': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-eik0n2aq': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-g7oeb4aw': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-tiabsvgr': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ykag01b2': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) /Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-nl2kdidn': [Errno 2] No such file or directory warnings.warn('resource_tracker: %r: %s' % (name, e)) --- 

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@JunyiXie@bedevere-bot@vstinner@shihai1991@the-knights-who-say-ni