gh-94440: Fix issue of ProcessPoolExecutor shutdown hanging#94468

yonatanp · 2022-06-30T21:22:45Z

Added a test to reproduce the issue and a fix to the issue itself.

Issue: ProcessPoolExecutor shutdown hangs after future cancel was requested #94440

ghost · 2022-06-30T21:22:48Z

All commit authors signed the Contributor License Agreement.

bedevere-bot · 2022-06-30T21:22:51Z

Most changes to Python require a NEWS entry.

Please add it using the blurb_it web app or the blurb command-line tool.

- documentation enhancement

Misc/NEWS.d/next/Library/2022-06-30-21-28-41.gh-issue-94440.LtgX0d.rst

AlexWaygood · 2023-03-14T22:03:33Z

@gpshead, would you be able to take a look at this PR?

AlexWaygood · 2023-03-15T16:28:39Z

(We prefer to avoid force pushes in the CPython repo, where possible -- they interact badly with the GitHub UI, making it hard to see what changed between commits. All PRs are squash-merged in CPython anyway, so there's no need to worry about having a messy commit history in this PR :)

yonatanp · 2023-03-15T16:31:51Z

(We prefer to avoid force pushes in the CPython repo, where possible -- they interact badly with the GitHub UI, making it hard to see what changed between commits. All PRs are squash-merged in CPython anyway, so there's no need to worry about having a messy commit history in this PR :)

Yeah - totally agree. I had the unfortunate case where my news change commit was created using my work-account, instead of my personal account. This in turn blocked the merge as the CLA Signing check would fail. I'm counting on the squash to clear it all out in the end :)

yonatanp · 2023-03-15T16:38:17Z

:sigh: I'm creating a bit of a mess with the news item thing.
I might need some expert help here to get it back on track.
The check of generated files up-to-date was failing, and I assumed it was because I edited the news item manually (where initially it was created by the blurb-it bot). So I created a new blurb with the bot, but now I have two news items. I will manually delete the new one and commit.
Still not sure what's then the issue with the failing "generated files" check.

AlexWaygood · 2023-03-15T16:40:33Z

Still not sure what's then the issue with the failing "generated files" check.

Probably just a fluke. If it happens again, ping me and I'll take a look.

The news entry looks good now!

yonatanp · 2023-03-15T16:51:47Z

Probably just a fluke. If it happens again, ping me and I'll take a look.

Failed again. It seems to be consistent.

AlexWaygood · 2023-03-15T16:55:26Z

Probably just a fluke. If it happens again, ping me and I'll take a look.
Failed again. It seems to be consistent.

It appears as though the CI check is broken somehow. It is failing on all PRs currently, e.g.:

Simplify and speed-up math.hypot() and math.dist() #102734

So: not your problem; don't worry about it :)

@ericsnowcurrently

The "check if generated files are up to date" CI check appears to be currently failing on all PRs (but not on pushes to main) See, for example: - python#94468 - python#94468 - python#102731 This appears to be because the C-analyzer tool doesn't like the `#line` directives introduced in python@70185de. I'm advised by the message printed to the terminal in https://github.com/python/cpython/actions/runs/4428706945/jobs/7768216988#step:14:84 that this is the appropriate short-term fix! Cc. @ericsnowcurrently

AlexWaygood · 2023-03-15T17:19:57Z

It appears as though the CI check is broken somehow. It is failing on all PRs currently, e.g.:

And here's (hopefully) a fix:

Exclude ceval.c from the C-analyzer tool #102735

The "check if generated files are up to date" CI check appears to be currently failing on all PRs (but not on pushes to main) See, for example: - #94468 - #94468 - #102731 This appears to be because the C-analyzer tool doesn't like the `#line` directives introduced in 70185de. I'm advised by the message printed to the terminal in https://github.com/python/cpython/actions/runs/4428706945/jobs/7768216988#step:14:84 that this is the appropriate short-term fix!

AlexWaygood · 2023-03-15T18:54:53Z

All green again now @yonatanp!

gpshead

Nice work debugging this. This looks good to me.

miss-islington · 2023-03-16T04:44:56Z

Thanks @yonatanp for the PR, and @gpshead for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

bedevere-bot · 2023-03-16T04:45:06Z

GH-102746 is a backport of this pull request to the 3.11 branch.

…thonGH-94468) Fix an issue of concurrent.futures ProcessPoolExecutor shutdown hanging. (cherry picked from commit 2dc9463) Co-authored-by: yonatanp <yonatan.perry@gmail.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

bedevere-bot · 2023-03-16T04:45:09Z

GH-102747 is a backport of this pull request to the 3.10 branch.

…thonGH-94468) Fix an issue of concurrent.futures ProcessPoolExecutor shutdown hanging. (cherry picked from commit 2dc9463) Co-authored-by: yonatanp <yonatan.perry@gmail.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

Fix an issue of concurrent.futures ProcessPoolExecutor shutdown hanging. (cherry picked from commit 2dc9463) Co-authored-by: yonatanp <yonatan.perry@gmail.com> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

* main: (34 commits) pythongh-102701: Fix overflow in dictobject.c (pythonGH-102750) pythonGH-78530: add support for generators in `asyncio.wait` (python#102761) Increase stack reserve size for Windows debug builds to avoid test crashes (pythonGH-102764) pythongh-102755: Add PyErr_DisplayException(exc) (python#102756) Fix outdated note about 'int' rounding or truncating (python#102736) pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (python#102760) pythongh-99726: Improves correctness of stat results for Windows, and uses faster API when available (pythonGH-102149) pythongh-102192: remove redundant exception fields from ssl module socket (python#102466) pythongh-102192: Replace PyErr_Fetch/Restore etc by more efficient alternatives (python#102743) pythongh-102737: Un-ignore ceval.c in the CI globals check (pythongh-102745) pythonGH-102748: remove legacy support for generator based coroutines from `asyncio.iscoroutine` (python#102749) pythongh-102721: Improve coverage of `_collections_abc._CallableGenericAlias` (python#102722) pythonGH-102653: Make recipe docstring show the correct distribution (python#102742) Add comments to `{typing,_collections_abc}._type_repr` about each other (python#102752) pythongh-102594: PyErr_SetObject adds note to exception raised on normalization error (python#102675) pythongh-94440: Fix issue of ProcessPoolExecutor shutdown hanging (python#94468) pythonGH-100112: avoid using iterable coroutines in asyncio internally (python#100128) pythongh-102690: Use Edge as fallback in webbrowser instead of IE (python#102691) pythongh-102660: Fix Refleaks in import.c (python#102744) pythongh-102738: remove from cases generator the code related to register instructions (python#102739) ...

The "check if generated files are up to date" CI check appears to be currently failing on all PRs (but not on pushes to main) See, for example: - python#94468 - python#94468 - python#102731 This appears to be because the C-analyzer tool doesn't like the `#line` directives introduced in python@70185de. I'm advised by the message printed to the terminal in https://github.com/python/cpython/actions/runs/4428706945/jobs/7768216988#step:14:84 that this is the appropriate short-term fix!

…thon#94468) Fix an issue of concurrent.futures ProcessPoolExecutor shutdown hanging. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

The "check if generated files are up to date" CI check appears to be currently failing on all PRs (but not on pushes to main) See, for example: - python#94468 - python#94468 - python#102731 This appears to be because the C-analyzer tool doesn't like the `#line` directives introduced in python@70185de. I'm advised by the message printed to the terminal in https://github.com/python/cpython/actions/runs/4428706945/jobs/7768216988#step:14:84 that this is the appropriate short-term fix!

…thon#94468) Fix an issue of concurrent.futures ProcessPoolExecutor shutdown hanging. Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>

pythongh-94440: Fix issue of ProcessPoolExecutor shutdown hanging
799d25c

bedevere-bot added the awaiting review label Jun 30, 2022

blurb-itbotand others added 2 commits June 30, 2022 21:28

📜🤖 Added by blurb_it.
1f444b2

pythongh-94440: Fix issue of ProcessPoolExecutor shutdown hanging
5e88846
- documentation enhancement

yonatanp mentioned this pull request Sep 20, 2022
ProcessPoolExecutor shutdown hangs after future cancel was requested #94440
Closed

AlexWaygood reviewed Mar 14, 2023
View reviewed changes

Misc/NEWS.d/next/Library/2022-06-30-21-28-41.gh-issue-94440.LtgX0d.rst Outdated Show resolvedHide resolved

AlexWaygood requested a review from gpshead March 14, 2023 22:03

Merge branch 'python:main' into fix-issue-94440
c376732

yonatanp force-pushed the fix-issue-94440 branch 2 times, most recently from eab6073 to c376732Compare March 15, 2023 16:22

rephrase news to describe user impact
061ccfb

📜🤖 Added by blurb_it.
b69bc4c

remove accidental extra news item created with blurb-it bot
46d1d48

AlexWaygood mentioned this pull request Mar 15, 2023
Exclude ceval.c from the C-analyzer tool #102735
Merged

AlexWaygood mentioned this pull request Mar 15, 2023
C-analyzer tool cannot parse #line directives #102737
Closed

Merge branch 'main' into fix-issue-94440
46be4f2

gpshead approved these changes Mar 16, 2023
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Mar 16, 2023

gpshead added needs backport to 3.10 only security fixes needs backport to 3.11 only security fixes labels Mar 16, 2023

gpshead merged commit 2dc9463 into python:mainMar 16, 2023

bedevere-bot removed the awaiting merge label Mar 16, 2023

bedevere-bot removed the needs backport to 3.11 only security fixes label Mar 16, 2023

bedevere-bot removed the needs backport to 3.10 only security fixes label Mar 16, 2023

gpshead self-assigned this Mar 16, 2023

Uh oh!

gh-94440: Fix issue of ProcessPoolExecutor shutdown hanging#94468

gh-94440: Fix issue of ProcessPoolExecutor shutdown hanging #94468

Uh oh!

Conversation

yonatanp commented Jun 30, 2022• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Jun 30, 2022• edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-bot commented Jun 30, 2022

Uh oh!

Uh oh!

AlexWaygood commented Mar 14, 2023

Uh oh!

AlexWaygood commented Mar 15, 2023

Uh oh!

yonatanp commented Mar 15, 2023

Uh oh!

yonatanp commented Mar 15, 2023

Uh oh!

AlexWaygood commented Mar 15, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yonatanp commented Mar 15, 2023

Uh oh!

AlexWaygood commented Mar 15, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexWaygood commented Mar 15, 2023

Uh oh!

AlexWaygood commented Mar 15, 2023

Uh oh!

gpshead left a comment

Choose a reason for hiding this comment

Uh oh!

miss-islington commented Mar 16, 2023

Uh oh!

bedevere-bot commented Mar 16, 2023

Uh oh!

bedevere-bot commented Mar 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yonatanp commented Jun 30, 2022•
edited by bedevere-bot
Loading

ghost commented Jun 30, 2022•
edited by ghost
Loading

AlexWaygood commented Mar 15, 2023•
edited
Loading

AlexWaygood commented Mar 15, 2023•
edited
Loading