bpo-29587: Enable implicit exception chaining with gen.throw()#19823

cjerdonek · 2020-04-30T21:43:45Z

This is a new version of PR #19811 to sort out the buildbot failures.

It enables implicit exception chaining when calling generator.throw(exc) by setting exc.__context__.

https://bugs.python.org/issue29587

Before this commit, if an exception was active inside a generator when calling gen.throw(), then that exception was lost (i.e. there was no implicit exception chaining). This commit fixes that.

Objects/genobject.c

vstinner

test_unittest still crash on this PR. I tested on Linux (Fedora 32).

$ make clean $ make $ ./python -m test -v test_unittest (...) test_patch_nested_autospec_repr (unittest.test.testmock.testpatch.PatchTest) ... ok test_patch_object_keyword_args (unittest.test.testmock.testpatch.PatchTest) ... ok test_patch_object_with_spec_as_boolean (unittest.test.testmock.testpatch.PatchTest) ... ok test_patch_orderdict (unittest.test.testmock.testpatch.PatchTest) ... ok test_patch_propagates_exc_on_exit (unittest.test.testmock.testpatch.PatchTest) ... Fatal Python error: Segmentation fault Current thread 0x00007f886bd1c740 (most recent call first): File "/home/vstinner/python/master/Lib/unittest/mock.py", line 1524 in __exit__ File "/home/vstinner/python/master/Lib/unittest/test/testmock/testpatch.py", line 1673 in __exit__ File "/home/vstinner/python/master/Lib/contextlib.py", line 498 in __exit__ File "/home/vstinner/python/master/Lib/unittest/mock.py", line 1323 in decoration_helper File "/home/vstinner/python/master/Lib/contextlib.py", line 135 in __exit__ File "/home/vstinner/python/master/Lib/unittest/mock.py", line 1337 in patched File "/home/vstinner/python/master/Lib/unittest/case.py", line 201 in handle File "/home/vstinner/python/master/Lib/unittest/case.py", line 733 in assertRaises File "/home/vstinner/python/master/Lib/unittest/test/testmock/testpatch.py", line 1692 in test_patch_propagates_exc_on_exit (...)

bedevere-bot · 2020-04-30T22:13:26Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

cjerdonek · 2020-04-30T22:26:51Z

Is there a way to mark this PR as "don't merge" for now? This is the same PR as before, so it has the same issue. The Azure CI job seems not to be running, which is what exhibited the failure before.

test_unittest still crash on this PR. I tested on Linux (Fedora 32).

Yes, this is the same failure as before. Everything passes locally on my Mac, so it will take me more time to look into this.

vstinner · 2020-04-30T22:51:14Z

Is there a way to mark this PR as "don't merge" for now?

Hum, I wasn't sure, so I did the 3 tricks that I know:

Add "do no merge label"
Add "[WIP]" in the PR title
Convert the PR to a draft

It seems like only the conversion to a draft technically prevents someone to merge the PR my mistake.

cjerdonek · 2020-04-30T22:52:08Z

Looks like that will cover it. ;)

cjerdonek · 2020-05-01T08:42:37Z

Okay, I believe my latest changes fix the buildbot failures from before.

Also, for future reference, below is a script I came up with that isolates the behavior difference between Mac and Fedora, with my prior version of the PR:

importreimportsysdeff(): exc, exc_value, tb=sys.exc_info() print(f'CLEARING FRAME: {tb.tb_frame!r}') tb.tb_frame.clear() defg(): # Uncommenting the following line caused the tb_frame.clear() line# above to exhibit the following platform-specific behavior:# 1) On Mac, this is logged to stderr# > TypeError: print_exception(): Exception expected for value,# NoneType found# 2) On Fedora 32, the following error happens:# > Fatal Python error: Segmentation faultdata=re.compile('xxx') try: yieldexceptException: f() gen=g() gen.send(None) gen.throw(ValueError)

Maybe this suggests an issue elsewhere in Python's code base.

Should this PR also get a What's New?

cjerdonek · 2020-05-01T08:47:01Z

I have made the requested changes; please review again.

bedevere-bot · 2020-05-01T08:47:04Z

Thanks for making the requested changes!

@vstinner: please review the changes made to this pull request.

vstinner

LGTM. "./python -m test -j0 -r" does no longer crash on my Fedora.

I understood that passing NULL value to _PyErr_ChainExceptions() created the bug? Maybe add an assertion to prevent the situation to happen again?

cjerdonek · 2020-05-01T13:33:21Z

Thank you.

Regarding an assertion, the thing is that it looks like _PyErr_ChainExceptions() is designed to accept a NULL value. So maybe in the future it can be made to work. (That's why I added an XXX comment.) So I wouldn't want to add an assertion making it seem like it always has to be like that. It could just something about how things are set up right now that was causing the crash.

Would you still like me to add an assertion and/or perhaps a code comment explaining this possibility?

bedevere-bot · 2020-05-02T01:14:22Z

@cjerdonek: Please replace # with GH- in the commit message next time. Thanks!

This is a follow-up to GH-19823 that removes the check that the exception value isn't NULL, prior to calling _PyErr_ChainExceptions(). This enables implicit exception chaining for gen.throw() in more circumstances. The commit also adds a test that a particular code snippet involving gen.throw() doesn't crash. The test shows why the new `gi_exc_state.exc_type != Py_None` check that was added is necessary. Without the new check, the code snippet (as well as a number of other tests) crashes on certain platforms (e.g. Fedora but not Mac).

cjerdonek · 2020-05-03T07:41:36Z

For future reference, I was able to remove the NULL check in a later PR (#19877): 21893fb

cjerdonek added 2 commits April 30, 2020 14:38

Add a failing test for issue 29587.
e757c76

bpo-29587: set exc.__context__ when calling gen.throw() (pythonGH-19811)
57597f2
Before this commit, if an exception was active inside a generator when calling gen.throw(), then that exception was lost (i.e. there was no implicit exception chaining). This commit fixes that.

cjerdonek requested a review from 1st1 as a code owner April 30, 2020 21:43

the-knights-who-say-ni added the CLA signed label Apr 30, 2020

bedevere-bot added the awaiting core review label Apr 30, 2020

1st1 reviewed Apr 30, 2020
View reviewed changes

Objects/genobject.c Outdated Show resolvedHide resolved

vstinner requested changes Apr 30, 2020
View reviewed changes

bedevere-bot removed the awaiting core review label Apr 30, 2020

bedevere-bot added the awaiting changes label Apr 30, 2020

vstinner added the DO-NOT-MERGE label Apr 30, 2020

vstinner changed the title ~~bpo-29587: Enable implicit exception chaining with gen.throw()~~[WIP] bpo-29587: Enable implicit exception chaining with gen.throw()Apr 30, 2020

vstinner marked this pull request as draft April 30, 2020 22:49

cjerdonek added 2 commits May 1, 2020 01:09

Wrap the long line.
c321731

Address the CI failures on Fedora, etc.
1bf3309

cjerdonek removed the DO-NOT-MERGE label May 1, 2020

cjerdonek marked this pull request as ready for review May 1, 2020 08:43

cjerdonek changed the title ~~[WIP] bpo-29587: Enable implicit exception chaining with gen.throw()~~bpo-29587: Enable implicit exception chaining with gen.throw()May 1, 2020

bedevere-bot added awaiting change review and removed awaiting changes labels May 1, 2020

bedevere-bot requested a review from vstinner May 1, 2020 08:47

vstinner approved these changes May 1, 2020
View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels May 1, 2020

cjerdonek merged commit 0204726 into python:masterMay 2, 2020

bedevere-bot removed the awaiting merge label May 2, 2020

cjerdonek deleted the fix-issue-29587-2 branch May 2, 2020 01:16

This was referenced May 2, 2020
bpo-29587: Enable exception chaining for gen.throw() with "yield from" #19858
Merged
bpo-29587: Add another test for the gen.throw() exception chain fix #19859
Merged
bpo-29587: Remove the exc_value NULL check in _gen_throw() #19877
Merged

Uh oh!

bpo-29587: Enable implicit exception chaining with gen.throw()#19823

bpo-29587: Enable implicit exception chaining with gen.throw() #19823

Uh oh!

Conversation

cjerdonek commented Apr 30, 2020• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-bot commented Apr 30, 2020

Uh oh!

cjerdonek commented Apr 30, 2020

Uh oh!

vstinner commented Apr 30, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjerdonek commented Apr 30, 2020

Uh oh!

cjerdonek commented May 1, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cjerdonek commented May 1, 2020

Uh oh!

bedevere-bot commented May 1, 2020

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

cjerdonek commented May 1, 2020

Uh oh!

bedevere-bot commented May 2, 2020

Uh oh!

cjerdonek commented May 3, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cjerdonek commented Apr 30, 2020•
edited by bedevere-bot
Loading

vstinner commented Apr 30, 2020•
edited
Loading

cjerdonek commented May 1, 2020•
edited
Loading