PEP 585 and ForwardRef

[wyfo]

BPO	41370
Nosy	@gvanrossum, @ericvsmith, @ambv, @ilevkivskyi, @JelleZijlstra, @miss-islington, @NiklasRosenstein, @isidentical, @sobolevn, @wyfo, @Fidget-Spinner, @AlexWaygood
PRs	bpo-41370: Add note about ForwardRefs and PEP585 generic types in docs #25183 [3.9] bpo-41370: Add note about ForwardRefs and PEP585 generic types in docs (GH-25183) #25184 bpo-41370: Evaluate strings as forward refs in PEP 585 generics #30900

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-07-22.20:16:14.238>
labels = ['type-bug', '3.9']
title = 'PEP 585 and ForwardRef'
updated_at = <Date 2022-03-07.18:03:08.022>
user = 'https://github.com/wyfo'

bugs.python.org fields:

activity = <Date 2022-03-07.18:03:08.022>
actor = 'gvanrossum'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2020-07-22.20:16:14.238>
creator = 'joperez'
dependencies = []
files = []
hgrepos = []
issue_num = 41370
keywords = ['patch']
message_count = 18.0
messages = ['374105', '374107', '374109', '374291', '390194', '390196', '409974', '410020', '411656', '411658', '411660', '411667', '411670', '411673', '411674', '411677', '411681', '414688']
nosy_count = 12.0
nosy_names = ['gvanrossum', 'eric.smith', 'lukasz.langa', 'levkivskyi', 'JelleZijlstra', 'miss-islington', 'n_rosenstein', 'BTaskaya', 'sobolevn', 'joperez', 'kj', 'AlexWaygood']
pr_nums = ['25183', '25184', '30900']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue41370'
versions = ['Python 3.9']

[wyfo]

PEP-585 current implementation (3.10.0a0) differs from current Generic implementation about ForwardRef, as illustrated bellow:

from dataclasses import dataclass, field
from typing import get_type_hints, List, ForwardRef

@dataclass
class Node:
 children: list["Node"] = field(default_factory=list)
 children2: List["Node"] = field(default_factory=list)

assert get_type_hints(Node) =={"children": list["Node"], "children2": List[Node]}
assert List["Node"].__args__ == (ForwardRef("Node"),)
assert list["Node"].__args__ == ("Node",) # No ForwardRef here, so no evaluation by get_type_hints

There is indeed no kind of ForwardRef for list arguments. As shown in the example, this affects the result of get_type_hints for recursive types handling.

He could be "fixed" in 2 lines in typing._eval_type with something like this :

def _eval_type(t, globalns, localns, recursive_guard=frozenset()):
 if isinstance(t, str):
 t = ForwardRef(t)
 if isinstance(t, ForwardRef):
 ...

but it's kind of hacky/dirty.

It's true that this issue will not concern legacy code, 3.9 still being not released. So developers of libraries using get_type_hints could add in their documentation that from __future__ import annotations is mandatory for recursive types with PEP-585 (I think I will do it).

By the way, Guido has quickly given his opinion about it in PR 21553: "We probably will not ever support this: importing ForwardRef from the built-in generic alias code would be problematic, and once from __future__ import annotations is always on there's no need to quote the argument anyway." (So feel free to close this issue)

[gvanrossum]

I think mentioning this in the docs is the best we can do in 3.9, and for 3.10 the point will be moot. The next release is release candidate 1, so we're well past the point where we can implement new functionality.

[wyfo]

However, PEP-563 will not solve the recursive type alias issue like A = list["A"] but this is a minor concern.

[gvanrossum]

Hm, recursive type aliases are an interesting issue. We may be able to do better there for 3.10, even if we can't fix it for 3.9 (or at least not for 3.9.0).

But in the meantime maybe you can come up with a PR that adds a note to the typing docs in 3.10 explaining that list["int"] will not be resolved to list[int], even though it works for List["int"]?

[gvanrossum]

New changeset 2b5913b by Ken Jin in branch 'master':
bpo-41370: Add note about ForwardRefs and PEP-585 generic types in docs (bpo-25183)
2b5913b

[miss-islington]

New changeset d56bcf9 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-41370: Add note about ForwardRefs and PEP-585 generic types in docs (GH-25183) (GH-25184)
d56bcf9

[NiklasRosenstein]

I'm running into this issue right now. Can anyone provide a rationale as to why you think this is acceptable/expected behaviour? Do we expect developers to semi-rely on get_type_hints(), but than still having to manually resolve forward references in PEP-585 generic aliases? That seems broken to me.

Thanks,
Niklas R.

[gvanrossum]

Niklas, can you show a brief example showing the issue you're running into? Is it just that list["Node"].__args__ is just ("Node",), not (ForwardRef("Node"),)? Or is it more complex?

[NiklasRosenstein]

Guido, sorry for the late response on this. I have a work around, but it involves passing along my own "context" from which to resolve strings on the go as they are encountered while decomposing the type hint.

NiklasRosenstein/python-databind@960da61

I'm using typing.get_type_hints() expecting it to fully resolve all forward references, but that no longer happens with PEP-585 generics.

https://github.com/NiklasRosenstein/databind/blob/960da61149b7139ec81b0e49d407fae321581914/databind.core/src/databind/core/types/utils.py#L129-L138

I understand the documentation has been updated to reflect this behaviour, but it was an issue for me because it broke it's original API contract.

In addition, forward references encoded as string literals are handled by evaluating them in globals and locals namespaces.

Arguably the same has happened when include_extras was added (Annotated was now stripped from the returned resolved type hints by default), but that had an easy fix by simply wrapping it with include_extra=True depending on the Python version. The fix for the forward references in PEP-585 was not so trivial because we can't hook into what get_type_hints() does when it encounters a string.

[gvanrossum]

I asked for a brief example that explains your issue to me. Instead you sent me links to production code and patches to it. Sorry, but that doesn't help me understand your problem. Is there really no way that you can write a little story that goes along the lines of

In our production code, we use the pattern

foo blah blah:
spam spam Ham eggs anchovies
buzz buzz blah get_type_hints buzz buzz

a lot. As you can see, the get_type_hints call fails
when we switch from "Ham eggs" to "ham eggs".

[NiklasRosenstein]

You're right, let me trim it down:

In production we use get_type_hints() a lot, expecting it to resolve strings as forward references as per it's original API contract. However, PEP-585 generics parametrized with strings in Python 3.10 doesn't work like that (as the documentation already points out). get_type_hints() itself does not fail, subsequently broke our code because it was not built to expect strings in GenericAlias.__args__.

What I ask myself is what motivated the decision to change the behaviour for PEP-585 generics in get_type_hints() and not go the extra mile.