fix(DONT MERGE): duckdb>=1.4.1 typing & warnings

[dangotbanned]

Will close #3188

What type of PR is this? (check all applicable)

• 💾 Refactor
• ✨ Feature
• 🐛 Bug Fix
• 🔧 Optimization
• 📝 Documentation
• ✅ Test
• 🐳 Other

Related issues

• Multiple `1.4.1` type stub issues duckdb/duckdb-python#124
• typing: Re-pin & fix duckdb==1.4.1 #3188
• Unpin duckdb in CI #3137
• DeprecationWarning: duckdb.typing is deprecated and will be removed in a future version. Please use duckdb.sqltypes instead. eakmanrq/sqlframe#531
• Question: How are non-narwhals-supported datatypes handled? akmalsoliev/Validoopsie#30

If you have comments or can explain your changes, please do so below

Note

Typing is "fixed"
Still need to open some issues/PRs upstream in duckdb (#3189 (comment))

[FBruzzesi]

Thanks @dangotbanned - I personally have mixed feeling for all the patching that is happening in the typing module. It's roughly 100 lines to take care of 6 issues that arguably duckdb should deal with?! Isn't there also the risk to lose track of potential future updates/improvement they might do just because we are taking care of all these details? I am no expert but I will give a try to simplifying it a bit

I am 1000% onboarded on all the remaining changes

[FBruzzesi]

Yeah I notice that as well! It's worse than having Any 😢

[dangotbanned]

Welp I found where this ends up (most recent last)

• https://github.com/duckdb/duckdb-python/blob/df7789cbd31b2d2b8d03d012f14331bc3297fb2d/src/duckdb_py/pyexpression.cpp#L348-L359
• https://github.com/duckdb/duckdb-python/blob/df7789cbd31b2d2b8d03d012f14331bc3297fb2d/src/duckdb_py/native/python_conversion.cpp#L1075-L1079
• https://github.com/duckdb/duckdb-python/blob/df7789cbd31b2d2b8d03d012f14331bc3297fb2d/src/duckdb_py/native/python_conversion.cpp#L916-L1069

Now I just need to work through TransformPythonObjectInternal to see what is allowed.
But it is definitely not Expression | str

https://github.com/duckdb/duckdb-python/blob/df7789cbd31b2d2b8d03d012f14331bc3297fb2d/_duckdb-stubs/__init__.pyi#L1015

[dangotbanned]

@FBruzzesi!!!! I wasn't expecting it to support all of this, but here's the proof 😂

from __future__ import annotations

import datetime as dt  # noqa: F811
import decimal
import uuid
from typing import Any, Callable, cast  # noqa: F811

import duckdb  # noqa: F811
import numpy as np  # noqa: F811
import pandas as pd

lit = cast("Callable[[Any], duckdb.Expression]", duckdb.ConstantExpression)


def try_lit(*inputs: Any) -> None:
    for value in inputs:
        print(f"{value}: {lit(value)}")


try_lit(
    None,
    pd.NaT,
    pd.NA,
    np.ma.masked,
    True,
    False,
    0,
    -999_991,
    100_000_000_000_000,
    1e98,
    decimal.Decimal("2933.957546"),
    uuid.uuid5(uuid.NAMESPACE_URL, "hello"),
    dt.datetime(2034, 1, 23, 4, 2, 56),
    dt.time(1, 2, 3, 4),
    dt.date(1023, 3, 1),
    dt.timedelta(50),
    "i am a string",
    bytearray(range(4)),
    memoryview(b"i was a bytestring"),
    b"i am a bytestring",
    ["list", "of", "str"],
    ("tuple", "of", "str"),
    {"dict": "str"},
    np.arange(10),
    np.datetime64(dt.date(2000, 3, 1)),
    duckdb.UnsignedIntegerValue(50),
)

None: NULL
NaT: NULL
<NA>: NULL
--: NULL
True: true
False: false
0: 0
-999991: -999991
100000000000000: 100000000000000
1e+98: 1e+98
2933.957546: 2933.957546
074171de-bc84-5ea4-b636-1135477620e1: '074171de-bc84-5ea4-b636-1135477620e1'::UUID
2034-01-23 04:02:56: '2034-01-23 04:02:56'::TIMESTAMP
01:02:03.000004: '01:02:03.000004'::TIME
1023-03-01: '1023-03-01'::DATE
50 days, 0:00:00: '50 days'::INTERVAL
i am a string: 'i am a string'
bytearray(b'\x00\x01\x02\x03'): '\x00\x01\x02\x03'::BLOB
<memory at 0x00000154FFEA3100>: 'i was a bytestring'::BLOB
b'i am a bytestring': 'i am a bytestring'::BLOB
['list', 'of', 'str']: ['list', 'of', 'str']
('tuple', 'of', 'str'): ['tuple', 'of', 'str']
{'dict': 'str'}: {'dict': 'str'}
[0 1 2 3 4 5 6 7 8 9]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
2000-03-01: '2000-03-01'::DATE
50: 50

[dangotbanned]

fix: It found a bug!!!

[dangotbanned]

I personally have mixed feeling for all the patching that is happening in the typing module. It's roughly 100 lines to take care of 6 issues that arguably duckdb should deal with?! Isn't there also the risk to lose track of potential future updates/improvement they might do just because we are taking care of all these details?

@FBruzzesi ahh I thought I was safe in draft 😂

I vaguely mentioned in a comment somewhere (that I can't seem to find 🤦‍♂️) about needing to upstream some stuff to duckdb.

All of (https://github.com/narwhals-dev/narwhals/pull/3189/files#diff-3e3b75f6b9584445d40b9b7564a703fb4399c24094039fef6d8ab041d43954c7) is essentially experimenting with what we need for most of typing to be "fixed" - so I can refer to it in an issue 🙂

I still need to work out the right signature for these:

• LambdaExpression
• ConstantExpression

I thought (once I've fixed them on our side) they might be an easier first issue to raise over there 🙏

[MarcoGorelli]

thanks for your pr

tbh i feel like this is too complex

once duckdb and ibis make a compatible release i'll make a pr showing a different idea

[dangotbanned]

#3189 (review)

Thanks for the review @MarcoGorelli, but I don't want to merge most of this to narwhals (#3189 (comment))

I'm just trying to gather things we currently would need to get the typing working - but I actually want to fix this upstream.
Those fixes might not be what is in here, but my hope is that this PR can serve to demonstrate that we're in need of some help from our upstream friends 😅

TL;DR: the new stubs don't match the implementation

[dangotbanned]

Superseded by #3433