fix(typing): Add overloads for ibis `lit` by dangotbanned · Pull Request #2972 · narwhals-dev/narwhals

dangotbanned · 2025-08-11T18:33:32Z

What type of PR is this? (check all applicable)

Related issues

Major culprit in Tracking: ibis typing fixes #2960

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

This is an example of a small, targeted upstream PR we could do.

For now, it covers a lot of our cases and makes the backend code easier to read - without needing to wait on acceptance elsewhere

Major culprit in #2960

dangotbanned · 2025-08-11T18:35:33Z

narwhals/_ibis/utils.py

+@overload
+def lit(value: str, dtype: None = ...) -> ir.StringScalar: ...
+@overload
+def lit(value: PythonLiteral | ir.Value, dtype: None = ...) -> ir.Scalar: ...


I've only included ir.Value here since our typing has a lot of PythonLiteral | ir.Value that doesn't get narrowed any further than that

FBruzzesi

Thanks @dangotbanned I only have one question :)

narwhals/_ibis/utils.py

FBruzzesi · 2025-08-13T19:29:50Z

narwhals/_ibis/utils.py

+@overload
+def lit(value: Literal[-1, 0, 1, 2, 3], dtype: None = ...) -> ir.IntegerScalar: ...
+@overload
+def lit(value: float, dtype: None = ...) -> ir.FloatingScalar | ir.IntegerScalar: ...


Disclaimer: I know nothing about ibis

Question: why wouldn't the intuitive mapping work? I tried locally and have no issue 🤔

Suggested change

@overload

def lit(value: Literal[-1, 0, 1, 2, 3], dtype: None = ...) -> ir.IntegerScalar: ...

@overload

def lit(value: float, dtype: None = ...) -> ir.FloatingScalar | ir.IntegerScalar: ...

@overload

def lit(value: int, dtype: None = ...) -> ir.IntegerScalar: ...

@overload

def lit(value: float, dtype: None = ...) -> ir.FloatingScalar: ...

Ah right so both this one and (#2972 (comment)) are to do with overlapping overloads

I might be able to do a repro later, but the general idea is:

in python's typing float means int | float (because reasons 😭)

in python's runtime, bool subclasses int (because PEP 285)

So I was trying to make use of Literal to avoid overlaps

Literal[-1, 0, 1, 2, 3] just so happened to cover all of our usage 😂

Ideally we'd have the more reasonable:

@overload def lit(value: bool, dtype: None = ...) -> ir.BooleanScalar: ... @overload def lit(value: int, dtype: None = ...) -> ir.IntegerScalar: ... @overload def lit(value: float, dtype: None = ...) -> ir.FloatingScalar: ...

Admittedly, I went straight to using Literal, as pyarrow-stubs don't do that - and it ends up bool being matched everywhere 😢

Which I tried to fix in (zen-xu/pyarrow-stubs#209)

@MarcoGorelli have you run into this issue with all your work on the numpy stubs?

A lot of the discussion I've seen on the int | float thing usually mention numpy-like use cases being a challenge

the overload you put first will get matched first, so you can ignore the overload overlap (like we do for DataFrame.__getitem__ with str vs Sequence[str], IIRC)

I might be able to do a repro later

Just finished putting this together, and it looks like we don't need to worry about this for ibis 😱

The problem is mainly with mypy, but I forgot that it doesn't understand ibis to begin with 😄:

Show HUGE REPRO narwhals/_ibis/t.py

from __future__ import annotations import datetime as dt from typing import TYPE_CHECKING, Any, Literal, overload import ibis from typing_extensions import reveal_type if TYPE_CHECKING: import ibis.expr.types as ir from typing_extensions import TypeAlias from narwhals.typing import PythonLiteral Incomplete: TypeAlias = Any """Marker for upstream issues.""" @overload def lit(value: Literal[False, True], dtype: None = ...) -> ir.BooleanScalar: ... @overload def lit(value: Literal[-1, 0, 1, 2, 3], dtype: None = ...) -> ir.IntegerScalar: ... @overload def lit(value: float, dtype: None = ...) -> ir.FloatingScalar | ir.IntegerScalar: ... @overload def lit(value: str, dtype: None = ...) -> ir.StringScalar: ... @overload def lit(value: PythonLiteral | ir.Value, dtype: None = ...) -> ir.Scalar: ... @overload def lit(value: Any, dtype: Any) -> Incomplete: ... def lit(value: Any, dtype: Any | None = None) -> Incomplete: """Alias for `ibis.literal`.""" literal: Incomplete = ibis.literal return literal(value, dtype) @overload def lit_intuitive(value: bool, dtype: None = ...) -> ir.BooleanScalar: ... # noqa: FBT001 @overload def lit_intuitive(value: int, dtype: None = ...) -> ir.IntegerScalar: ... @overload def lit_intuitive(value: float, dtype: None = ...) -> ir.FloatingScalar: ... @overload def lit_intuitive(value: str, dtype: None = ...) -> ir.StringScalar: ... @overload def lit_intuitive(value: PythonLiteral | ir.Value, dtype: None = ...) -> ir.Scalar: ... @overload def lit_intuitive(value: Any, dtype: Any) -> Incomplete: ... def lit_intuitive(value: Any, dtype: Any | None = None) -> Incomplete: literal: Incomplete = ibis.literal return literal(value, dtype) def here_we_go() -> None: # noqa: PLR0914, PLR0915 # ruff: noqa: E741 a: Literal[False] = False b: Literal[True] = True c: Literal[False, True] = True d: bool = False e: Literal[0] = 0 f: Literal[1] = 1 g: Literal[-1, 0, 1, 2, 3] = -1 h: int = 1 i: int = 4 j: float = 0.0 k: str = "k" l: PythonLiteral = 1 m: dt.date = dt.date(2000, 1, 1) n: Any = 0 a_1 = lit(a) a_2 = lit_intuitive(a) reveal_type(a_1), reveal_type(a_2) b_1 = lit(b) b_2 = lit_intuitive(b) reveal_type(b_1), reveal_type(b_2) c_1 = lit(c) c_2 = lit_intuitive(c) reveal_type(c_1), reveal_type(c_2) d_1 = lit(d) d_2 = lit_intuitive(d) reveal_type(d_1), reveal_type(d_2) e_1 = lit(e) e_2 = lit_intuitive(e) reveal_type(e_1), reveal_type(e_2) f_1 = lit(f) f_2 = lit_intuitive(f) reveal_type(f_1), reveal_type(f_2) g_1 = lit(g) g_2 = lit_intuitive(g) reveal_type(g_1), reveal_type(g_2) h_1 = lit(h) h_2 = lit_intuitive(h) reveal_type(h_1), reveal_type(h_2) i_1 = lit(i) i_2 = lit_intuitive(i) reveal_type(i_1), reveal_type(i_2) j_1 = lit(j) j_2 = lit_intuitive(j) reveal_type(j_1), reveal_type(j_2) k_1 = lit(k) k_2 = lit_intuitive(k) reveal_type(k_1), reveal_type(k_2) l_1 = lit(l) l_2 = lit_intuitive(l) reveal_type(l_1), reveal_type(l_2) m_1 = lit(m) m_2 = lit_intuitive(m) reveal_type(m_1), reveal_type(m_2) n_1 = lit(n) n_2 = lit_intuitive(n) reveal_type(n_1), reveal_type(n_2)

Outputs

mypy

>>> mypy narwhals/_ibis/t.py narwhals/_ibis/t.py:74: note: Revealed type is "Any" narwhals/_ibis/t.py:78: note: Revealed type is "Any" narwhals/_ibis/t.py:82: note: Revealed type is "Any" narwhals/_ibis/t.py:86: note: Revealed type is "Any" narwhals/_ibis/t.py:90: note: Revealed type is "Any" narwhals/_ibis/t.py:94: note: Revealed type is "Any" narwhals/_ibis/t.py:98: note: Revealed type is "Any" narwhals/_ibis/t.py:102: note: Revealed type is "Any" narwhals/_ibis/t.py:106: note: Revealed type is "Any" narwhals/_ibis/t.py:110: note: Revealed type is "Any" narwhals/_ibis/t.py:114: note: Revealed type is "Any" narwhals/_ibis/t.py:118: note: Revealed type is "Any" narwhals/_ibis/t.py:122: note: Revealed type is "Any" narwhals/_ibis/t.py:126: note: Revealed type is "Any" Success: no issues found in 1 source file

pyright

>>> pyright narwhals/_ibis/t.py narwhals/_ibis/t.py narwhals/_ibis/t.py:74:17 - information: Type of "a_1" is "BooleanScalar" narwhals/_ibis/t.py:74:35 - information: Type of "a_2" is "BooleanScalar" narwhals/_ibis/t.py:78:17 - information: Type of "b_1" is "BooleanScalar" narwhals/_ibis/t.py:78:35 - information: Type of "b_2" is "BooleanScalar" narwhals/_ibis/t.py:82:17 - information: Type of "c_1" is "BooleanScalar" narwhals/_ibis/t.py:82:35 - information: Type of "c_2" is "BooleanScalar" narwhals/_ibis/t.py:86:17 - information: Type of "d_1" is "BooleanScalar" narwhals/_ibis/t.py:86:35 - information: Type of "d_2" is "BooleanScalar" narwhals/_ibis/t.py:90:17 - information: Type of "e_1" is "IntegerScalar" narwhals/_ibis/t.py:90:35 - information: Type of "e_2" is "IntegerScalar" narwhals/_ibis/t.py:94:17 - information: Type of "f_1" is "IntegerScalar" narwhals/_ibis/t.py:94:35 - information: Type of "f_2" is "IntegerScalar" narwhals/_ibis/t.py:98:17 - information: Type of "g_1" is "IntegerScalar" narwhals/_ibis/t.py:98:35 - information: Type of "g_2" is "IntegerScalar" narwhals/_ibis/t.py:102:17 - information: Type of "h_1" is "IntegerScalar" narwhals/_ibis/t.py:102:35 - information: Type of "h_2" is "IntegerScalar" narwhals/_ibis/t.py:106:17 - information: Type of "i_1" is "FloatingScalar | IntegerScalar" narwhals/_ibis/t.py:106:35 - information: Type of "i_2" is "IntegerScalar" narwhals/_ibis/t.py:110:17 - information: Type of "j_1" is "FloatingScalar | IntegerScalar" narwhals/_ibis/t.py:110:35 - information: Type of "j_2" is "FloatingScalar" narwhals/_ibis/t.py:114:17 - information: Type of "k_1" is "StringScalar" narwhals/_ibis/t.py:114:35 - information: Type of "k_2" is "StringScalar" narwhals/_ibis/t.py:118:17 - information: Type of "l_1" is "IntegerScalar" narwhals/_ibis/t.py:118:35 - information: Type of "l_2" is "IntegerScalar" narwhals/_ibis/t.py:122:17 - information: Type of "m_1" is "Scalar" narwhals/_ibis/t.py:122:35 - information: Type of "m_2" is "Scalar" narwhals/_ibis/t.py:126:17 - information: Type of "n_1" is "Unknown" narwhals/_ibis/t.py:126:35 - information: Type of "n_2" is "Unknown" 0 errors, 0 warnings, 28 informations

Important
pyright does resolve i: int and j: float to different overloads!!! 🥳

ibis doesn't have py.typed so mypy ignores it completely afaiu

(#2972 (comment)), (#2972 (comment)) See repro (#2972 (comment)) Co-authored-by: FBruzzesi <francesco.bruzzesi.93@gmail.com>

FBruzzesi

Neat! Thanks for the adjustment!

dangotbanned added 2 commits August 11, 2025 18:22

fix(typing): Add overloads for ibis lit

cf9b19a

Major culprit in #2960

fix(typing): breathing clearly now

147c228

dangotbanned added internal typing ibis Issue is related to ibis backend labels Aug 11, 2025

dangotbanned commented Aug 11, 2025

View reviewed changes

dangotbanned marked this pull request as ready for review August 11, 2025 18:45

dangotbanned added 3 commits August 12, 2025 18:15

Merge branch 'main' into ibis-typing-fixes-1

9785230

Merge branch 'main' into ibis-typing-fixes-1

1816175

Merge branch 'main' into ibis-typing-fixes-1

90a789e

FBruzzesi reviewed Aug 13, 2025

View reviewed changes

dangotbanned and others added 3 commits August 14, 2025 10:51

Merge remote-tracking branch 'upstream/main' into ibis-typing-fixes-1

efe2e4f

refactor(typing): Simplify overloads

9fea0a7

(#2972 (comment)), (#2972 (comment)) See repro (#2972 (comment)) Co-authored-by: FBruzzesi <francesco.bruzzesi.93@gmail.com>

Merge branch 'main' into ibis-typing-fixes-1

ba3d7ee

dangotbanned requested a review from FBruzzesi August 14, 2025 12:51

FBruzzesi approved these changes Aug 14, 2025

View reviewed changes

dangotbanned merged commit 5ea6ee4 into main Aug 14, 2025
30 of 31 checks passed

dangotbanned deleted the ibis-typing-fixes-1 branch August 14, 2025 13:42

dangotbanned mentioned this pull request Aug 8, 2025

Tracking: ibis typing fixes #2960

Open

8 tasks

dangotbanned mentioned this pull request Oct 20, 2025

Tracking: Enabling more pyright rules #3103

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(typing): Add overloads for ibis `lit`#2972

fix(typing): Add overloads for ibis `lit`#2972
dangotbanned merged 8 commits intomainfrom
ibis-typing-fixes-1

dangotbanned commented Aug 11, 2025

Uh oh!

dangotbanned Aug 11, 2025

Uh oh!

FBruzzesi left a comment

Uh oh!

Uh oh!

FBruzzesi Aug 13, 2025

Uh oh!

dangotbanned Aug 13, 2025 •

edited

Loading

Uh oh!

dangotbanned Aug 13, 2025

Uh oh!

MarcoGorelli Aug 14, 2025

Uh oh!

dangotbanned Aug 14, 2025

Uh oh!

MarcoGorelli Aug 14, 2025

Uh oh!

dangotbanned Aug 14, 2025

Uh oh!

FBruzzesi left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

dangotbanned commented Aug 11, 2025

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

Uh oh!

dangotbanned Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

FBruzzesi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FBruzzesi Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

dangotbanned Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dangotbanned Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

MarcoGorelli Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

dangotbanned Aug 14, 2025

Choose a reason for hiding this comment

Outputs

mypy

pyright

Uh oh!

MarcoGorelli Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

dangotbanned Aug 14, 2025

Choose a reason for hiding this comment

Uh oh!

FBruzzesi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dangotbanned Aug 13, 2025 •

edited

Loading

FBruzzesi left a comment •

edited

Loading