Unable to use concurrent.futures in atexit hook

[jd]

BPO	42647
Nosy	@pitrou, @vstinner, @jd, @aeros, @iritkatriel

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-12-15.13:59:10.547>
labels = ['3.9', 'type-crash']
title = 'Unable to use concurrent.futures in atexit hook'
updated_at = <Date 2021-02-01.13:41:43.911>
user = 'https://github.com/jd'

bugs.python.org fields:

activity = <Date 2021-02-01.13:41:43.911>
actor = 'corona10'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2020-12-15.13:59:10.547>
creator = 'jd'
dependencies = []
files = []
hgrepos = []
issue_num = 42647
keywords = []
message_count = 4.0
messages = ['383058', '385584', '385972', '386067']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'jd', 'aeros', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue42647'
versions = ['Python 3.9']

[jd]

Python 3.9 introduced a regression with concurrent.futures. The following program works fine on Python < 3.8 but raises an error on 3.9:

import atexit
import concurrent.futures

def spawn():
 with concurrent.futures.ThreadPoolExecutor() as t:
 pass

atexit.register(spawn)

$ python3.9 rep.py
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
 File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
 File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
 File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
 File "<frozen importlib._bootstrap_external>", line 790, in exec_module
 File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
 File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 37, in <module>
 threading._register_atexit(_python_exit)
 File "/usr/local/Cellar/python@3.9/3.9.0_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 1370, in _register_atexit
 raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown

[iritkatriel]

b61b818 is the first bad commit
commit b61b818
Author: Kyle Stanley <aeros167@gmail.com>
Date: Fri Mar 27 15:31:22 2020 -0400

bpo-39812: Remove daemon threads in concurrent.futures (GH-19149)

Remove daemon threads from :mod:`concurrent.futures` by adding
an internal `threading._register_atexit()`, which calls registered functions
prior to joining all non-daemon threads. This allows for compatibility
with subinterpreters, which don't support daemon threads.

[aeros]

Thanks for bringing attention to this, Julien.

While the regression is definitely unfortunate, I'm uncertain about whether the behavior is *correct* to allow in the first place. Specifically, allowing the registration of an atexit hook which uses a ThreadPoolExecutor within it means that the finalization of the executor will be done *after* thread finalization occurs, leaving dangling threads which will have to be abruptly killed upon interpreter exit instead of being safely joined. From my perspective at least, this doesn't seem like something to be encouraged.

Is there a real-world situation where it's specifically necessary or even beneficial to utilize ThreadPoolExecutor at this point after thread finalization rather than earlier in the program? Not that it doesn't exist, but to me it intuitively seems very odd to utilize an executor within an atexit hook, which are intended to just be resource finalization/cleanup functions called at interpreter shutdown. Assuming there is a genuine use case I'm not seeing, it may be worth weighing against the decision to convert the executors to not use daemon threads, as I presently don't think there's a way to (safely) allow that behavior without reverting back to using daemon threads.

That said, I'll admit that I'm more than a bit biased as the author of the commit which introduced the regression, so I'll CC Antoine Pitrou (active expert for threading and concurrent.futures) to help make the final decision.

[jd]

Is there a real-world situation where it's specifically necessary or even beneficial to utilize ThreadPoolExecutor at this point after thread finalization rather than earlier in the program? Not that it doesn't exist, but to me it intuitively seems very odd to utilize an executor within an atexit hook, which are intended to just be resource finalization/cleanup functions called at interpreter shutdown. Assuming there is a genuine use case I'm not seeing, it may be worth weighing against the decision to convert the executors to not use daemon threads, as I presently don't think there's a way to (safely) allow that behavior without reverting back to using daemon threads.

To put that in perspective, here is the original issue that trigged this bug for me:

Traceback (most recent call last):
 File "/root/project/ddtrace/profiling/scheduler.py", line 50, in flush
 exp.export(events, start, self._last_export)
 File "/root/project/ddtrace/profiling/exporter/http.py", line 186, in export
 self._upload(client, self.endpoint_path, body, headers)
 File "/root/project/ddtrace/profiling/exporter/http.py", line 189, in _upload
 self._retry_upload(self._upload_once, client, path, body, headers)
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
 do = self.iter(retry_state=retry_state)
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter
 return fut.result()
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 433, in result
 return self.__get_result()
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
 raise self._exception
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__
 result = fn(*args, **kwargs)
 File "/root/project/ddtrace/profiling/exporter/http.py", line 193, in _upload_once
 client.request("POST", path, body=body, headers=headers)
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1255, in request
 self._send_request(method, url, body, headers, encode_chunked)
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1301, in _send_request
 self.endheaders(body, encode_chunked=encode_chunked)
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1250, in endheaders
 self._send_output(message_body, encode_chunked=encode_chunked)
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 1010, in _send_output
 self.send(msg)
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 950, in send
 self.connect()
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/http/client.py", line 921, in connect
 self.sock = self._create_connection(
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/socket.py", line 88, in create_connection
 addrs = list(getaddrinfo(host, port, 0, SOCK_STREAM))
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_socketcommon.py", line 247, in getaddrinfo
 addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags)
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 841, in _get_resolver
 self._resolver = self.resolver_class(hub=self) # pylint:disable=not-callable
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/resolver/thread.py", line 39, in __init__
 self.pool = hub.threadpool
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 865, in _get_threadpool
 self._threadpool = self.threadpool_class(self.threadpool_size, hub=self)
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/hub.py", line 860, in threadpool_class
 return GEVENT_CONFIG.threadpool
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 50, in getter
 return self.settings[setting_name].get()
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 146, in get
 self.value = self.validate(self._default())
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 248, in validate
 return self._import_one_of([self.shortname_map.get(x, x) for x in value])
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 223, in _import_one_of
 return self._import_one(candidates[-1])
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/_config.py", line 237, in _import_one
 module = importlib.import_module(module)
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/importlib/__init__.py", line 127, in import_module
 return _bootstrap._gcd_import(name[level:], package, level)
 File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
 File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
 File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
 File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
 File "<frozen importlib._bootstrap_external>", line 790, in exec_module
 File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
 File "/root/project/.tox/py39-profile-gevent/lib/python3.9/site-packages/gevent/threadpool.py", line 748, in <module>
 class ThreadPoolExecutor(concurrent.futures.ThreadPoolExecutor):
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/__init__.py", line 49, in __getattr__
 from .thread import ThreadPoolExecutor as te
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/thread.py", line 37, in <module>
 threading._register_atexit(_python_exit)
 File "/root/.pyenv/versions/3.9.0/lib/python3.9/threading.py", line 1370, in _register_atexit
 raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown

What's happening is that the ddtrace library registers an atexit hook that does an HTTP call. As the application runs using gevent, some gevent modules that were not loaded are loaded during the atexit() hook and the loading of concurrent.futures.thread is done very late, at the point where the interpreter is shutting down.

I'm totally fine blaming gevent here if you prefer. The problem is that there's nothing preventing any library call to be made in an atexit() hook, and any library could decide to use concurent.futures without the library user being able to do anything about it, except maybe, making sure concurent.futures is loaded very early in the program. However, having to load this library even if you don't use it to be sure it does not break would break the separation of principles.

At at least, at this stage, it might be the responsibility of Python to make sure all threading._register_at_exit calls are done whatever if the library is used or not (i.e. that'd mean loading concurent.futures with threading unconditionally).

[jamesra]

I'm running into the same issue after attempting to migrate my code to Python 3.10. My use case is there is a temporary folder full of files. I have an atexit hook which deletes the temporary folder recursively in parallel. The temporary folder is over a network mount and shutil.rmtree runs too slowly. I'm far from a python expert but it seems very odd to remove API functionality simply because a notification has been triggered. I'm not really sure how to work around this other than registering a separate "atexit safe" rmtree implementation. How is a user supposed to know which calls are safe to use in atexit and which are not?

In case it is informative this is the implementation for my use-case. I register this function to be called by atexit. (Credit goes to a stackoverflow post I read sometime trying to figure out why shutil.rmtree was so slow)

atexit.register(rmtree, _sharedTempRoot)

def rmtree(directory): 
 with concurrent.futures.ThreadPoolExecutor() as executor:
 for root, dirs, files in os.walk(directory, topdown=False):
 try: 
 files_futures = executor.map(os.remove, [os.path.join(root, f) for f in files])
 t = list(files_futures) #Force the map operation to complete
 except Exception as e:
 prettyoutput.error(f'{e}')
 
 try:
 dir_futures = executor.map(os.rmdir, [os.path.join(root, d) for d in dirs])
 d = list(dir_futures) #Force the map operation to complete
 except Exception as e:
 prettyoutput.error(f'{e}')
 
 os.rmdir(directory)
 
 return