Skip to content

bpo-41559: Documentation for PEP 612#24000

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gvanrossum merged 9 commits into from
Jan 2, 2021
Merged

bpo-41559: Documentation for PEP 612 #24000

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
18d63f2
docs for pep612
Fidget-Spinner Dec 29, 2020
a2fc0d1
polish up docs
Fidget-Spinner Dec 30, 2020
bebe250
remove repetitive statements
Fidget-Spinner Dec 30, 2020
2e13c99
polish flow
Fidget-Spinner Dec 30, 2020
ece68b8
Merge remote-tracking branch 'upstream/master' into docs-pep612
Fidget-Spinner Dec 30, 2020
21f9958
use suggestions
Fidget-Spinner Dec 31, 2020
2ea0761
fix up problems
Fidget-Spinner Dec 31, 2020
4eb116b
small nit
Fidget-Spinner Dec 31, 2020
f314a34
Merge remote-tracking branch 'upstream/master' into docs-pep612
Fidget-Spinner Jan 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions Doc/library/stdtypes.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -4959,6 +4959,11 @@ All parameterized generics implement special read-only attributes.
(~T,)


.. note::
A ``GenericAlias`` object with :class:`typing.ParamSpec` parameters may not
have correct ``__parameters__`` after substitution because
:class:`typing.ParamSpec` is intended primarily for static type checking.
Copy link
Member

@gvanrossumgvanrossumDec 30, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to add this to the note, but privately I'm interested in an example where it goes wrong?

Copy link
MemberAuthor

@Fidget-SpinnerFidget-SpinnerDec 31, 2020•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

'''This example is from the typing module, though it will behave the same with the builtin GenericAlias'''fromtypingimportParamSpec, TypeVar, GenericP=ParamSpec('P') T=TypeVar('T') classX(Generic[P]): ... # Empty tuple, because T is nested inside that listprint(X[int, str, T].__parameters__)


.. seealso::

* :pep:`585` -- "Type Hinting Generics In Standard Collections"
Expand Down
204 changes: 203 additions & 1 deletion Doc/library/typing.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,8 @@
--------------

This module provides runtime support for type hints as specified by
:pep:`484`, :pep:`526`, :pep:`544`, :pep:`586`, :pep:`589`, :pep:`591`, and :pep:`613`.
:pep:`484`, :pep:`526`, :pep:`544`, :pep:`586`, :pep:`589`, :pep:`591`,
:pep:`612` and :pep:`613`.
The most fundamental support consists of the types :data:`Any`, :data:`Union`,
:data:`Tuple`, :data:`Callable`, :class:`TypeVar`, and
:class:`Generic`. For full specification please see :pep:`484`. For
Expand DownExpand Up@@ -171,6 +172,22 @@ It is possible to declare the return type of a callable without specifying
the call signature by substituting a literal ellipsis
for the list of arguments in the type hint: ``Callable[..., ReturnType]``.

Callables which take other callables as arguments may indicate that their
parameter types are dependent on each other using :class:`ParamSpec`.
Additionally, if that callable adds or removes arguments from other
callables, the :data:`Concatenate` operator may be used. They
take the form ``Callable[ParamSpecVariable, ReturnType]`` and
``Callable[Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable], ReturnType]``
respectively.

.. versionchanged:: 3.10
``Callable`` now supports :class:`ParamSpec` and :data:`Concatenate`.
See :pep:`612` for more information.

.. seealso::
The documentation for :class:`ParamSpec` and :class:`Concatenate` provide
examples of usage in ``Callable``.

.. _generics:

Generics
Expand DownExpand Up@@ -316,6 +333,43 @@ User defined generic type aliases are also supported. Examples::
.. versionchanged:: 3.7
:class:`Generic` no longer has a custom metaclass.

User-defined generics for parameter expressions are also supported via parameter
specification variables in the form ``Generic[P]``. The behavior is consistent
with type variables' described above as parameter specification variables are
treated by the typing module as a specialized type variable. The one exception
to this is that a list of types can be used to substitute a :class:`ParamSpec`::

>>> from typing import Generic, ParamSpec, TypeVar

>>> T = TypeVar('T')
>>> P = ParamSpec('P')

>>> class Z(Generic[T, P]): ...
...
>>> Z[int, [dict, float]]
__main__.Z[int, (<class 'dict'>, <class 'float'>)]


Furthermore, a generic with only one parameter specification variable will accept
parameter lists in the forms ``X[[Type1, Type2, ...]]`` and also
``X[Type1, Type2, ...]`` for aesthetic reasons. Internally, the latter is converted
to the former and are thus equivalent::

>>> class X(Generic[P]): ...
...
>>> X[int, str]
__main__.X[(<class 'int'>, <class 'str'>)]
>>> X[[int, str]]
__main__.X[(<class 'int'>, <class 'str'>)]

Do note that generics with :class:`ParamSpec` may not have correct
``__parameters__`` after substitution in some cases because they
are intended primarily for static type checking.

.. versionchanged:: 3.10
:class:`Generic` can now be parameterized over parameter expressions.
See :class:`ParamSpec` and :pep:`612` for more details.

A user-defined generic class can have ABCs as base classes without a metaclass
conflict. Generic metaclasses are not supported. The outcome of parameterizing
generics is cached, and most types in the typing module are hashable and
Expand DownExpand Up@@ -602,10 +656,80 @@ These can be used as types in annotations using ``[]``, each having a unique syn
``Callable[..., Any]``, and in turn to
:class:`collections.abc.Callable`.

Callables which take other callables as arguments may indicate that their
parameter types are dependent on each other using :class:`ParamSpec`.
Additionally, if that callable adds or removes arguments from other
callables, the :data:`Concatenate` operator may be used. They
take the form ``Callable[ParamSpecVariable, ReturnType]`` and
``Callable[Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable], ReturnType]``
respectively.

.. deprecated:: 3.9
:class:`collections.abc.Callable` now supports ``[]``. See :pep:`585` and
:ref:`types-genericalias`.

.. versionchanged:: 3.10
``Callable`` now supports :class:`ParamSpec` and :data:`Concatenate`.
See :pep:`612` for more information.

.. seealso::
The documentation for :class:`ParamSpec` and :class:`Concatenate` provide
examples of usage with ``Callable``.

.. data:: Concatenate

Used with :data:`Callable` and :class:`ParamSpec` to type annotate a higher
order callable which adds, removes, or transforms parameters of another
callable. Usage is in the form
``Concatenate[Arg1Type, Arg2Type, ..., ParamSpecVariable]``. ``Concatenate``
is currently only valid when used as the first argument to a :data:`Callable`.
The last parameter to ``Concatenate`` must be a :class:`ParamSpec`.

For example, to annotate a decorator ``with_lock`` which provides a
:class:`threading.Lock` to the decorated function, ``Concatenate`` can be
used to indicate that ``with_lock`` expects a callable which takes in a
``Lock`` as the first argument, and returns a callable with a different type
signature. In this case, the :class:`ParamSpec` indicates that the returned
callable's parameter types are dependent on the parameter types of the
callable being passed in::

from collections.abc import Callable
from threading import Lock
from typing import Any, Concatenate, ParamSpec

P = ParamSpec('P')
R = ParamSpec('R')

# Use this lock to ensure that only one thread is executing a function
# at any time.
my_lock = Lock()

def with_lock(f: Callable[Concatenate[Lock, P], R]) -> Callable[P, R]:
'''A type-safe decorator which provides a lock.'''
global my_lock
def inner(*args: P.args, **kwargs: P.kwargs) -> T:
# Provide the lock as the first argument.
return f(my_lock, *args, **kwargs)
return inner

@with_lock
def sum_threadsafe(lock: Lock, numbers: list[float]) -> float:
'''Add a list of numbers together in a thread-safe manner.'''
with lock:
return sum(numbers)

# We don't need to pass in the lock ourselves thanks to the decorator.
sum_threadsafe([1.1, 2.2, 3.3])

.. versionadded:: 3.10

.. seealso::

* :pep:`612` -- Parameter Specification Variables (the PEP which introduced
``ParamSpec`` and ``Concatenate``).
* :class:`ParamSpec` and :class:`Callable`.


.. class:: Type(Generic[CT_co])

A variable annotated with ``C`` may accept a value of type ``C``. In
Expand DownExpand Up@@ -876,6 +1000,84 @@ These are not used in annotations. They are building blocks for creating generic
for the type variable must be a subclass of the boundary type,
see :pep:`484`.

.. class:: ParamSpec(name, *, bound=None, covariant=False, contravariant=False)

Parameter specification variable. A specialized version of
:class:`type variables <TypeVar>`.

Usage::

P = ParamSpec('P')

Parameter specification variables exist primarily for the benefit of static
type checkers. They are used to forward the parameter types of one
callable to another callable -- a pattern commonly found in higher order
functions and decorators. They are only valid when used in ``Concatenate``,
or as the first argument to ``Callable``, or as parameters for user-defined
Generics. See :class:`Generic` for more information on generic types.

For example, to add basic logging to a function, one can create a decorator
``add_logging`` to log function calls. The parameter specification variable
tells the type checker that the callable passed into the decorator and the
new callable returned by it have inter-dependent type parameters::

from collections.abc import Callable
from typing import TypeVar, ParamSpec
import logging

T = TypeVar('T')
P = ParamSpec('P')

def add_logging(f: Callable[P, T]) -> Callable[P, T]:
'''A type-safe decorator to add logging to a function.'''
def inner(*args: P.args, **kwargs: P.kwargs) -> T:
logging.info(f'{f.__name__} was called')
return f(*args, **kwargs)
return inner

@add_logging
def add_two(x: float, y: float) -> float:
'''Add two numbers together.'''
return x + y

Without ``ParamSpec``, the simplest way to annotate this previously was to
use a :class:`TypeVar` with bound ``Callable[..., Any]``. However this
causes two problems:

1. The type checker can't type check the ``inner`` function because
``*args`` and ``**kwargs`` have to be typed :data:`Any`.
2. :func:`~cast` may be required in the body of the ``add_logging``
decorator when returning the ``inner`` function, or the static type
checker must be told to ignore the ``return inner``.

.. attribute:: args
.. attribute:: kwargs

Since ``ParamSpec`` captures both positional and keyword parameters,
``P.args`` and ``P.kwargs`` can be used to split a ``ParamSpec`` into its
components. ``P.args`` represents the tuple of positional parameters in a
given call and should only be used to annotate ``*args``. ``P.kwargs``
represents the mapping of keyword parameters to their values in a given call,
and should be only be used to annotate ``**kwargs`` or ``**kwds``. Both
attributes require the annotated parameter to be in scope.

Parameter specification variables created with ``covariant=True`` or
``contravariant=True`` can be used to declare covariant or contravariant
generic types. The ``bound`` argument is also accepted, similar to
:class:`TypeVar`. However the actual semantics of these keywords are yet to
be decided.

.. versionadded:: 3.10

.. note::
Only parameter specification variables defined in global scope can
be pickled.

.. seealso::
* :pep:`612` -- Parameter Specification Variables (the PEP which introduced
``ParamSpec`` and ``Concatenate``).
* :class:`Callable` and :class:`Concatenate`.

.. data:: AnyStr

``AnyStr`` is a type variable defined as
Expand Down
22 changes: 22 additions & 0 deletions Doc/whatsnew/3.10.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -144,6 +144,28 @@ See :pep:`604` for more details.

(Contributed by Maggie Moss and Philippe Prados in :issue:`41428`.)

PEP 612: Parameter Specification Variables
------------------------------------------

Two new options to improve the information provided to static type checkers for
:pep:`484`\ 's ``Callable`` have been added to the :mod:`typing` module.

The first is the parameter specification variable. They are used to forward the
parameter types of one callable to another callable -- a pattern commonly
found in higher order functions and decorators. Examples of usage can be found
in :class:`typing.ParamSpec`. Previously, there was no easy way to type annotate
dependency of parameter types in such a precise manner.

The second option is the new ``Concatenate`` operator. It's used in conjunction
with parameter specification variables to type annotate a higher order callable
which adds or removes parameters of another callable. Examples of usage can
be found in :class:`typing.Concatenate`.

See :class:`typing.Callable`, :class:`typing.ParamSpec`,
:class:`typing.Concatenate` and :pep:`612` for more details.

(Contributed by Ken Jin in :issue:`41559`.)

Other Language Changes
======================

Expand Down
15 changes: 8 additions & 7 deletions Lib/typing.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -544,8 +544,9 @@ def TypeAlias(self, parameters):

@_SpecialForm
def Concatenate(self, parameters):
"""Used in conjunction with ParamSpec and Callable to represent a higher
order function which adds, removes or transforms parameters of a Callable.
"""Used in conjunction with ``ParamSpec`` and ``Callable`` to represent a
higher order function which adds, removes or transforms parameters of a
callable.

For example::

Expand DownExpand Up@@ -735,11 +736,11 @@ class ParamSpec(_Final, _Immutable, _TypeVarLike, _root=True):

Parameter specification variables exist primarily for the benefit of static
type checkers. They are used to forward the parameter types of one
Callable to another Callable, a pattern commonly found in higher order
functions and decorators. They are only valid when used in Concatenate, or
as the first argument to Callable, or as parameters for user-defined Generics.
See class Generic for more information on generic types. An example for
annotating a decorator::
callable to another callable, a pattern commonly found in higher order
functions and decorators. They are only valid when used in ``Concatenate``,
or s the first argument to ``Callable``, or as parameters for user-defined
Generics. See class Generic for more information on generic types. An
example for annotating a decorator::

T = TypeVar('T')
P = ParamSpec('P')
Expand Down