Simplify the interpreter's (type, val, tb) exception representation

[iritkatriel]

BPO	45711
Nosy	@gvanrossum, @terryjreedy, @scoder, @vstinner, @markshannon, @brandtbucher, @da-woods, @iritkatriel
PRs	bpo-45711: Use _PyErr_ClearExcState instead of setting only exc_value… #29404 bpo-45711: remove unnecessary DUP_TOP and POP in exception handling #29495 bpo-45711: assert that the type of exc_info is redundant #29518 bpo-45711: Re-bump the magic number and update doc #29528 bpo-45711: use exc_value instead of exc_type to determine if exc_info is valid. Add more assertions. #29627 bpo-45711: Change exc_info related APIs to derive type and traceback from the exception instance #29780 bpo-45711: [asyncio] Normalize exceptions immediately after Fetch, before they are stored as StackItem, which should be normalized #29890 bpo-45711: Remove unnecessary normalization of exc_info #29922 bpo-45711: Remove type and traceback from exc_info #30122 bpo-46219, 46221: simplify except* implementation following exc_info changes. Move helpers to exceptions.c. Do not assume that exception groups are truthy. #30289 bpo-45711: move whatsnew entries which are incorrectly listed under N… #30849

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-13.19:33:16.899>
created_at = <Date 2021-11-04.11:29:47.242>
labels = ['interpreter-core', '3.11', 'performance']
title = "Simplify the interpreter's (type, val, tb) exception representation"
updated_at = <Date 2022-03-13.19:33:16.899>
user = 'https://github.com/iritkatriel'

bugs.python.org fields:

activity = <Date 2022-03-13.19:33:16.899>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-13.19:33:16.899>
closer = 'iritkatriel'
components = ['Interpreter Core']
creation = <Date 2021-11-04.11:29:47.242>
creator = 'iritkatriel'
dependencies = []
files = []
hgrepos = []
issue_num = 45711
keywords = ['patch']
message_count = 31.0
messages = ['405681', '405839', '405849', '406114', '406120', '406209', '406238', '406983', '406984', '407402', '407597', '408043', '408768', '408786', '409453', '411497', '412264', '412265', '412270', '412272', '412273', '412275', '412281', '412289', '412290', '412292', '412303', '412304', '412316', '414482', '415086']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'terry.reedy', 'scoder', 'vstinner', 'Mark.Shannon', 'brandtbucher', 'da-woods', 'iritkatriel']
pr_nums = ['29404', '29495', '29518', '29528', '29627', '29780', '29890', '29922', '30122', '30289', '30849']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue45711'
versions = ['Python 3.11']

[iritkatriel]

Exceptions are represented in the interpreter as (type, val, tb) triplets which most of the time contain redundant information (the type is the type of val and the tb is also on the exception). This complicates the code and is inefficient as opcodes that manage exceptions push and pop 3 items for each exception.

We will change the internal representation to be (1) just the exception value if it is normalised and (2) a tuple of the 3 values for the uncommon case where they are all needed.

See also faster-cpython/ideas#106.

[terryjreedy]

Would there be any change at the Python level?

[iritkatriel]

Initially not, neither in python nor in the c api.

It would be nice to replace PyErr_Fetch/Restore by a version that takes just an exception but that’s a long deprecation.

[iritkatriel]

New changeset 05fbd60 by Irit Katriel in branch 'main':
bpo-45711: Use _PyErr_ClearExcState instead of setting only exc_value to NULL (GH-29404)
05fbd60

[markshannon]

New changeset 4cdeee5 by Irit Katriel in branch 'main':
bpo-45711: remove unnecessary DUP_TOP and POP in exception handling (GH-29495)
4cdeee5

[iritkatriel]

New changeset 8f1b71d by Brandt Bucher in branch 'main':
bpo-45711: Re-bump the magic number and update doc (GH-29528)
8f1b71d

[iritkatriel]

New changeset de3db14 by Irit Katriel in branch 'main':
bpo-45711: assert that the type of exc_info is redundant (GH-29518)
de3db14

[iritkatriel]

New changeset c456dfa by Irit Katriel in branch 'main':
bpo-45711: use exc_value instead of exc_type to determine if exc_info is valid. Add more assertions. (GH-29627)
c456dfa

[iritkatriel]

Following the analysis/discussion on faster-cpython/ideas#106:

• exc_info is always normalized, so we actually will never need to create a tuple of these three values (exc_curinfo, the exception raised but not yet caught can be unnormalized, but it is not pushed/popped on the stack).

• We will reduce the interpreter's exc_info representation to just the exception instance.

• There are two APIs that are impacted, both in non-documented edge cases:

1. sys.exc_info()[2] can currently be different from sys.exc_info()[1].__traceback__ because changes to the latter (while an except clause is executing) don't show up in the former. This is arguably a bug, we will change it so that the type and traceback are always consistent with the exception.

2. PyErr_SetExcInfo does no arg checking, and will set exc_info to an inconsistent triplet if you ask it to. However, the exc_value arg must be an exception instance so the only thing you can do is pass in nonsensical args where the type/traceback do not match the exception. This function's purpose is to save/restore exc_info. We will make it ignore the type and traceback and document that change.

[iritkatriel]

New changeset 8a45ca5 by Irit Katriel in branch 'main':
bpo-45711: Change exc_info related APIs to derive type and traceback from the exception instance (GH-29780)
8a45ca5

[iritkatriel]

New changeset 2ff758b by Irit Katriel in branch 'main':
bpo-45711: [asyncio] Normalize exceptions immediately after Fetch, before they are stored as StackItem, which should be normalized (GH-29890)
2ff758b

[iritkatriel]

New changeset 2109f78 by Irit Katriel in branch 'main':
bpo-45711: Remove unnecessary normalization of exc_info (GH-29922)
2109f78