Rework starargs with union argument

[randolf-scholz]

Makes *args inference smarter by special casing UnionType

1. *(tuple[A, B, C] | tuple[None|None|None]) gets treated like tuple[A | None, B | None, C | None] inside ArgTypeExpander (applies if all union member are fixed size tuples of equal length
2. *(Iterable[X] | Iterable[Y]) gets treated like *Iterable[X | Y] inside ArgTypeExpander (applies if ① is not triggered)

This gives some better inference in some cases:

x: list[str|None] |list[str] reveal_type([*x]) # master: list[Any] PR: list[str | None]x2: tuple[int, int] |tuple[None, None] reveal_type( (*x2,) ) # master: tuple[Any, ...] PR: tuple[int | None, int | None]x3: tuple[int, int] |tuple[None, None, None] reveal_type( (*x3,) ) # master: tuple[Any, ...] PR: tuple[int | None, ...]

See added unit tests for more examples.

• Fixes: mypy fails to correctly understand some iterable type in *args #19662
• Fixes: Mypy fails to solve tuple[int, *Ts, int] as an Iterable[T]. #19659
• Fixes: Variadic *args where args: Union[Tuple[...], Tuple[...], ...] are _never_ checked [False Negative] #17161
• Fixes: Unpacking an iterable in a list comprehension leads to the type inferred as list[Any] #15747
• Fixes: Custom __iter__ method yields false positive when used with star args (*) #14470
• Fixes: Unpacking unions of tuples skips type checking #15969

See Also: #19650, cc @hauntsaninja

---

Dev notes:

• If we detect that the union has non-tuple elements or tuples of different sizes, we upcast-reinterpret every union member as some Iterable[T], then apply each union member, and then return the union of the results.
  Handling unions of finite size tuples with different sizes in infeasible, due to combinatoric explosion¹

• map_actuals_to_formals added special case for union of same sized tuples.

• ExpressionChecker.check_arg: added some extra logic that applies if the callee_type is UnpackType, but the actual_type got expanded from Unpack_type to something else. Previously, this only worked "by accident" because the expander was guaranteed to return AnyType(TypeOfAny.from_errors) in this case.

• ArgTypeExpander:

  • Create a NewType alias IterableType for Iteralble[T], and helper functions is_iterable, is_iterable_instance_type, _make_iterable_instance_type.
  • added _solve_as_iterable function that uses the solver to interpret a type as Iterable[T]
  • added as_iterable_type that applies a few special cases (UnionType, etc. before calling the solver)
  • added parse_star_args_type that converts the argument of *args to one of TupleType | IterableType | ParamSpecType | AnyType

• visit_tuple_expr: Applies ArgTypeExpander(*parse_star_args_type) to ensure consistent behavior across [*args], {*args} and (*args).

Footnotes

1. consider f(*x1, *x2, ..., *xn). If each $x_k$ is comprised of a union of $m_k$ differently sized tuples, then there are $m_1⋅m_2⋅…⋅m_k$ possible paths. ↩

[randolf-scholz]

Hm there is still some issue to be solved with Unpack. The flatten_nested_tuples method helps./

I think one could determine the correct upcast by using the solver, but I discovered some inconsistencies: #19652

[randolf-scholz]

This really does catch quite a few false negatives, for example:

Repro of homeassistant/components/govee_light_localhttps://mypy-play.net/?mypy=latest&python=3.12&gist=3bf7d56a2bfbc1fc23be62ef521dc613

defset_rgb_color(red: int, green: int, blue: int) ->None: ... _last_color_state: tuple[ str|None, int|None, tuple[int, int, int] |tuple[int|None] |None, ] color_mode, brightness, color=_last_color_stateifcolor: set_rgb_color(*color) # MASTER: no error ❌, PR: raises [arg-type] ✅

Repro of xarray/tests/test_namedarray: https://mypy-play.net/?mypy=latest&python=3.12&gist=9999ca2cb89e849978c979c1e4cc9bca

fromtypingimportIterable, Hashable type _Dim=Hashable type _Dims=tuple[_Dim, ...] type _DimsLike=str|Iterable[_Dim] defpermute_dims(*dim: Iterable[_Dim]) ->None: ... deftest(dims: _DimsLike) ->None: permute_dims(*dims) # master: no error ❌, PR: raises [arg-type] ✅

Repro of src/bokeh/server/tornado.py:450: https://mypy-play.net/?mypy=latest&python=3.12&gist=0df919e6986b3fcbdb9d7c8727fb8d48

fromtypingimportAnyclassRequestHandler: ... type RouteContext=dict[str, Any] type URLRoutes=list[ tuple[str, type[RequestHandler]] |tuple[str, type[RequestHandler], RouteContext], ] defdemo( all_patterns: URLRoutes, extra_patterns: URLRoutes, toplevel_patterns: URLRoutes, prefix: str, data: dict[str, Any], ) ->None: forpinextra_patterns+toplevel_patterns: prefixed_pat= (prefix+p[0], *p[1:], data) all_patterns.append(prefixed_pat) # MASTER: false negative, PR: [arg-type]

[randolf-scholz]

I think this is ready for initial review.

[randolf-scholz]

Found 4 more older issues that all get fixed:

• Fixes: Variadic *args where args: Union[Tuple[...], Tuple[...], ...] are _never_ checked [False Negative] #17161
• Fixes: Unpacking an iterable in a list comprehension leads to the type inferred as list[Any] #15747
• Fixes: Custom __iter__ method yields false positive when used with star args (*) #14470
• Fixes: Unpacking unions of tuples skips type checking #15969

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

tornado (https://github.com/tornadoweb/tornado) + tornado/routing.py:355: error: Argument 2 to "Rule" has incompatible type "*Union[list[Any], tuple[Any], tuple[Any, dict[str, Any]], tuple[Any, dict[str, Any], str]]"; expected "Optional[dict[str, Any]]" [arg-type]+ tornado/routing.py:355: error: Argument 2 to "Rule" has incompatible type "*Union[list[Any], tuple[Any], tuple[Any, dict[str, Any]], tuple[Any, dict[str, Any], str]]"; expected "Optional[str]" [arg-type]+ tornado/routing.py:357: error: Argument 1 to "Rule" has incompatible type "*Union[list[Any], tuple[Union[str, Matcher], Any], tuple[Union[str, Matcher], Any, dict[str, Any]], tuple[Union[str, Matcher], Any, dict[str, Any], str]]"; expected "Matcher" [arg-type]+ tornado/routing.py:357: error: Argument 1 to "Rule" has incompatible type "*Union[list[Any], tuple[Union[str, Matcher], Any], tuple[Union[str, Matcher], Any, dict[str, Any]], tuple[Union[str, Matcher], Any, dict[str, Any], str]]"; expected "Optional[dict[str, Any]]" [arg-type]+ tornado/routing.py:357: error: Argument 1 to "Rule" has incompatible type "*Union[list[Any], tuple[Union[str, Matcher], Any], tuple[Union[str, Matcher], Any, dict[str, Any]], tuple[Union[str, Matcher], Any, dict[str, Any], str]]"; expected "Optional[str]" [arg-type] hydpy (https://github.com/hydpy-dev/hydpy) + hydpy/core/selectiontools.py:370: error: Argument 1 to "intersection" of "Devices" has incompatible type "*Elements"; expected "Elements" [arg-type]+ hydpy/core/pubtools.py:163: error: Argument 1 to "Timegrids" has incompatible type "*Timegrids | Timegrid | tuple[datetime | str | Date, datetime | str | Date, timedelta | str | Period]"; expected "Timegrid" [arg-type]+ hydpy/core/pubtools.py:163: error: Argument 1 to "Timegrids" has incompatible type "*Timegrids | Timegrid | tuple[datetime | str | Date, datetime | str | Date, timedelta | str | Period]"; expected "Timegrid | None" [arg-type] aiohttp (https://github.com/aio-libs/aiohttp) + aiohttp/web_urldispatcher.py:671:16: error: Exception type must be derived from BaseException (or be a tuple of exception classes) [misc] colour (https://github.com/colour-science/colour) + colour/plotting/models.py:1986: error: Argument 2 to "_linear_equation" has incompatible type "*ndarray[tuple[int], dtype[float64]]"; expected "ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]" [arg-type] core (https://github.com/home-assistant/core) + homeassistant/components/govee_light_local/light.py:216: error: Argument 2 to "set_rgb_color" of "GoveeLocalApiCoordinator" has incompatible type "*tuple[int, int, int] | tuple[int | None]"; expected "int" [arg-type]+ homeassistant/components/govee_light_local/light.py:218: error: Argument 2 to "set_temperature" of "GoveeLocalApiCoordinator" has incompatible type "*tuple[int, int, int] | tuple[int | None]"; expected "int" [arg-type] comtypes (https://github.com/enthought/comtypes) - comtypes/_memberspec.py:85: error: Unused "type: ignore" comment [unused-ignore] werkzeug (https://github.com/pallets/werkzeug) + src/werkzeug/test.py:436: error: Argument 2 to "add_file" of "FileMultiDict" has incompatible type "*Union[tuple[IO[bytes], str], tuple[IO[bytes], str, str]]"; expected "Optional[str]" [arg-type] dd-trace-py (https://github.com/DataDog/dd-trace-py) + ddtrace/llmobs/_integrations/bedrock_agents.py:237: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "type[BaseException]" [arg-type]+ ddtrace/llmobs/_integrations/bedrock_agents.py:237: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "BaseException" [arg-type]+ ddtrace/llmobs/_integrations/bedrock_agents.py:286: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "type[BaseException]" [arg-type]+ ddtrace/llmobs/_integrations/bedrock_agents.py:286: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "BaseException" [arg-type]+ ddtrace/llmobs/_experiment.py:367: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "type[BaseException]" [arg-type]+ ddtrace/llmobs/_experiment.py:367: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "BaseException" [arg-type] dedupe (https://github.com/dedupeio/dedupe) + dedupe/core.py:196: error: Argument 1 to "zip" has incompatible type "*tuple[int, Mapping[str, Any]] | tuple[str, Mapping[str, Any]]"; expected "Iterable[str]" [arg-type] materialize (https://github.com/MaterializeInc/materialize) + misc/python/materialize/mzcompose/composition.py:382: error: Argument 1 to "Popen" has incompatible type "list[object]"; expected "str | bytes | PathLike[str] | PathLike[bytes] | Sequence[str | bytes | PathLike[str] | PathLike[bytes]]" [arg-type]+ misc/python/materialize/mzcompose/composition.py:433: error: Not all union combinations were tried because there are too many unions [misc]+ misc/python/materialize/mzcompose/composition.py:434: error: Argument 1 to "run" has incompatible type "list[object]"; expected "str | bytes | PathLike[str] | PathLike[bytes] | Sequence[str | bytes | PathLike[str] | PathLike[bytes]]" [arg-type] ignite (https://github.com/pytorch/ignite) + ignite/engine/engine.py:156: error: Argument 1 to "register_events" of "Engine" has incompatible type "*type[Events]"; expected "list[str] | list[EventEnum]" [arg-type]+ ignite/contrib/engines/tbptt.py:120: error: Argument 1 to "register_events" of "Engine" has incompatible type "*type[Tbptt_Events]"; expected "list[str] | list[EventEnum]" [arg-type] xarray (https://github.com/pydata/xarray) + xarray/tests/test_namedarray.py: note: In member "test_permute_dims" of class "TestNamedArray":+ xarray/tests/test_namedarray.py:545: error: Argument 1 to "permute_dims" of "NamedArray" has incompatible type "*str | Iterable[Hashable]"; expected "Iterable[Hashable] | EllipsisType" [arg-type]+ xarray/tests/test_namedarray.py: note: In class "TestNamedArray": bokeh (https://github.com/bokeh/bokeh) + src/bokeh/server/tornado.py: note: In member "__init__" of class "BokehTornado":+ src/bokeh/server/tornado.py:450:41: error: Argument 1 to "append" of "list" has incompatible type "tuple[dict[str, bool | dict[str, ApplicationContext] | str | None] | str | type[RequestHandler] | dict[str, Any], ...]"; expected "tuple[str, type[RequestHandler]] | tuple[str, type[RequestHandler], dict[str, Any]]" [arg-type]+ src/bokeh/server/tornado.py:453:37: error: Argument 1 to "append" of "list" has incompatible type "tuple[dict[str, bool | dict[str, ApplicationContext] | str | None] | str | type[RequestHandler] | dict[str, Any], ...]"; expected "tuple[str, type[RequestHandler]] | tuple[str, type[RequestHandler], dict[str, Any]]" [arg-type]+ src/bokeh/server/tornado.py: note: At top level:

[randolf-scholz]

The src/werkzeug/test.py is a case where the PR introduces a false positive: https://mypy-play.net/?mypy=latest&python=3.12&gist=eb790692cdcb803373269187d9ad0a4a

fromtypingimportIOdefadd_file(file: str|IO[bytes], filename: str|None=None) ->None: ... deftest(values: tuple[IO[bytes]] |tuple[IO[bytes], str]) ->None: add_file(*values) # master: OK ✅, PR: ❌

Because it upcasts the type to typing.Iterable[typing.IO[builtins.bytes] | builtins.str], whereas master treats it like tuple[Any, ...].

So we need to deal with the case when each union member is a finite tuple differently

1. One option would be to infer tuple[Any, ...] in this case like on master;
2. A bit better would be to use the equal size combination logic for the minimal length, and then the join for the tail part, so for instance in the example above, tuple[IO[bytes], *tuple[str, ...]] would be more precise than just tuple[Any, ...].
3. One could even theoretically do something like tuple[IO[bytes], *(tuple[()] | tuple[str])]], but this is outside the current typing spec, as it breaks the assumption that the argument to Unpack is a tuple or a TypeVarTuple.

I think I can do option 2 for now and combine it with the equal size tuple case.

[hauntsaninja]

thanks for picking this up!

[hauntsaninja]

can we reuse logic from analyze_iterable_item_type_without_expression or similar?

[sterliakov]

Can mypy.infer.infer_function_type_arguments replace this and following line at once?

[sterliakov]

AFAIC we only use fallback to store builtins.tuple or a NamedTuple class, never anything else, and there might be an implicit assumption elsewhere that fallback is assignable to tuple[Any, ...], for example. Will anything break if you use builtins.tuple Instance here instead?

[randolf-scholz]

tuple would be more appropriate; the issue is that currently ArgumentInferContext only knows about Mapping, Iterable and builtins.function.
Adding builtins.tuple there is possible, but will have to refactor all the fixtures to include a compatible tuple implementation.

[sterliakov]

#19662 is listed as closed by this PR, is this comment up to date? (the bug in the original implementation that caused #19662 was checking for and actual_type.args - that adds nothing useful and obviously breaks for non-generic subclasses, removing this check should suffice)

[sterliakov]

If you refactor L449-L458 into a helper function, this method would look like

... size = _precise_tuple_len(typ) if size is None: return False return all(_precise_tuple_len(t) == size for t in iterator) 

[sterliakov]

...or? :)

[sterliakov]

There are a couple more affected/fixed tickets from the scan above: #18320, #16492, #6799

[sterliakov]

A bit better would be to use the equal size combination logic for the minimal length, and then the join for the tail part

This sounds good to me if it can be trivially implemented. If not, let's leave that part as-is or use the any fallback (your option 1) and defer to a separate PR, this PR is already quite long.

[randolf-scholz]

@sterliakov@hauntsaninja So an update on this. I turned out this whole thing was way more difficult to fix holistically than anticipated.

The main issue is that when we want to deal with unions of tuples in a sensible manner, we already need to know the expected length of the argument when computing formal_to_actual.

However, inside the argmap.map_actuals_to_formals, we do not necessarily have access to the TypeInfo of typing.Iterable and builtins.tuple, which this PR uses to determine the actual length.

I managed to work around this, building machinery that computes an abstract representation without actually materializing the Tuple type associated with a star argument.

The resulting PR currently sits in my fork: randolf-scholz#6

I put quite a lot of effort into this, and I think the written code has generally high quality. But it has bloated in size to a monster PR (+3,838 −908) since I had to rewrite quite a few methods from scratch.

What do you advise?

[ilevkivskyi]

@randolf-scholz could you please clarify what do you mean by TypeInfos being not available in map_actuals_to_formals()? I would expect that all callers (essentially expression checker + few others) should have access to those TypeInfos, so you should be able to pass them as an additional argument there. Is it not the case for some reason?

[randolf-scholz]

@ilevkivskyi I'd need something like infer.ArgumentInferContext inside map_actuals_to_formals that has access to the TypeInfo of: typing.Iterable, builtins.tuple and builtins.function.

There are a few places where map_actuals_to_formals is used, and it is unclear to me how I'd get access to these TypeInfo:

• suggestions.py
• testinfer.py
• common.py
• stats.py (via map_formals_to_actuals)
• maybe a few others I haven't found

At least in functools.py I could use ctx.api.named_generic_type to get them. To be honest, it feels a bit weird that certain types¹ that are used by core language syntax are not always available, e.g. with a global variable or something, but need to be passed around with helpers like ArgumentInferContext.

Footnotes

1. That would be

   • builtins.tuple, since any call that expects *args converts these to builtins.tuple
   • builtins.dict, since any call that expects **kwargs converts these to builtins.list
   • builtins.list, since *args = ... unpacks these into a list
   • builtins.{tuple/list/set/dict} due to comprehensions
   • builtins.{None, bool, Ellipsis} due to the constants
   • builtins.slice due to slice syntax x[1:].
   • builtins.type, since match Foo() checks that Foo is a type.
   • collections.abc.Sequence, since case [a, b, c]: checks subclassing of that ABC
   • collections.abc.Mapping, since case{"text": message }: checks subclassing of that ABC
   • collections.abc.Iterable, since star-unpacking checks subtyping of that protocol.
   • SupportsKeysAndGetItem, since double star-unpacking checks subtyping of that protocol.
   • maybe I missed one or two.
   ↩

[ilevkivskyi]

To be honest, it feels a bit weird that certain types that are used by core language syntax are not always available.

Yes, we (most likely me) are going to change this at some point. But it may be not for another couple months. So you can either wait until then, or "pipe through" required TypeInfos via callers to each of those call sites.

[ilevkivskyi]

TBH I don't like the nested imports. I know it may be hard, but I would recommend to try finding a way without introducing new import cycles.

[randolf-scholz]

@ilevkivskyi

To be honest, it feels a bit weird that certain types that are used by core language syntax are not always available.
Yes, we (most likely me) are going to change this at some point. But it may be not for another couple months. So you can either wait until then, or "pipe through" required TypeInfos via callers to each of those call sites.

Well, #20472 is up now which solves these issues as I said by computing an abstract "Tuple Normal Form", that allows computing prefix and suffix length without access to infer.ArgumentInferContext. But that PR has become extremely large, so reviewing it will be challenging, and I am not sure how to cut it up into smaller bits.

This PR is probably dead unless these essential types become available globally.

[ilevkivskyi]

Well, yeah, that is huge. I guess the best way forward may be to simply wait until TypeInfos become globally available.