bpo-29842: Make Executor.map less eager so it handles large/unbounded…#18566
Uh oh!
There was an error while loading. Please reload this page.
Conversation
… input iterables appropriately
…lds no reference to result at the moment it yields Reduce line lengths to PEP8 limits
rdarder left a comment
rdarder left a comment There was a problem hiding this comment.
Hi there! sorry for intruding. I was looking for this feature and decided to try this patch out. Works great! Just found a small issue so I sent you a comment.
Thank you!
| except StopIteration: | ||
| argsiter = None | ||
| else: | ||
| fs.append(self.submit(fn, *args)) |
There was a problem hiding this comment.
If the executor has been shut down, this will raise:cannot schedule new futures after shutdown
But, also the base executor holds no state, so at this level it'll be pretty hard to tell if the executor has been shut down or not.
There was a problem hiding this comment.
@rdarder
this could catch that RuntimeError and just return, or the executor could keep a weakset of result_iterators and close them on shutdown
Doc/library/concurrent.futures.rst Outdated
| .. versionchanged:: 3.5 | ||
| Added the *chunksize* argument. | ||
| .. versionchanged:: 3.9 |
There was a problem hiding this comment.
| .. versionchanged:: 3.9 | |
| .. versionchanged:: 3.11 |
Uh oh!
There was an error while loading. Please reload this page.
Tronic commented Oct 29, 2021
While waiting for this to be included in Python, one can easily copy&paste the map function out of this version and call it with the executor as the first argument. Hoping that you get this merged soon! |
MojoVampire added performance 
Uh oh!
There was an error while loading. Please reload this page.
kumaraditya303 added type-feature 
@graingert Tests are failing, can you fix them and rebase to main? I'll then review and run through the buildbots. |
Misc/NEWS.d/next/Library/2019-05-06-12-49-21.bpo-29842.Tw-ETh.rst Outdated Show resolvedHide resolved
Uh oh!
There was an error while loading. Please reload this page.
Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>
Uh oh!
There was an error while loading. Please reload this page.
| yield fs.pop().result() | ||
| res = fs.popleft().result() | ||
| else: | ||
| yield fs.pop().result(end_time - time.monotonic()) |
There was a problem hiding this comment.
this doesn't cancel the currently waited on fut correctly: #95166
importcontextlibimportfunctoolsimportconcurrent.futuresimportthreadingimportsysdeffn(num, stop_event): ifnum==1: stop_event.wait() return"done 1"ifnum==2: return"done 2"defmain(): stop_event=threading.Event() log= [] withconcurrent.futures.ThreadPoolExecutor(max_workers=1) aspool: defprint_n_wait(ident): log.append(f"{ident=} started") try: stop_event.wait() finally: log.append(f"{ident=} stopped") fut=pool.submit(print_n_wait, ident="first") try: withcontextlib.closing(pool.map(print_n_wait, ["second", "third"], timeout=1)) asgen: try: next(gen) exceptconcurrent.futures.TimeoutError: print("timed out") else: raiseRuntimeError("timeout expected") finally: stop_event.set() assertlog== ["ident='first' started", "ident='first' stopped"], f"{log=} is wrong"if__name__=="__main__": sys.exit(main())kumaraditya303 commented Jul 23, 2022
@graingert You can run |
Jason-Y-Z commented Feb 3, 2024
Hey all, I took the inspiration from this PR and continued the work in #114975 |
Hi, fyi here is a follow up PR: #125663 🙏🏻 |
10 participants
… input iterables appropriately
bugs.python.org/issue29842
bugs.python.org/issue29842
recreate of #707 with conflicts fixed and versionchanged updated to py3.9
/cc @MojoVampire
https://bugs.python.org/issue29842