Make `import_optional_dependency` util

[MarcoGorelli]

We should make such a util, which can take an optional argument specifying the minimum version. Then we can raise unified and consistent error messages

[mikeweltevrede]

Sounds interesting @MarcoGorelli - could you explain a bit more on this?

[MarcoGorelli]

hey @mikeweltevrede ! good to hear from you again!

we have some places where we import optional dependencies inline, e.g.

narwhals/narwhals/dataframe.py

Lines 503 to 507 in 5f1f19f

	try:
	import pyarrow as pa # ignore-banned-import
	except ModuleNotFoundError as exc: # pragma: no cover
	msg = f"PyArrow>=14.0.0 is required for `DataFrame.__arrow_c_stream__` for object of type {type(native_frame)}"
	raise ModuleNotFoundError(msg) from exc

We should probably have a helper function which does this, and a custom narwhals error for missing optional dependencies

pandas has something similar (import_optional_dependency in their source code) which we could probably use (with attribution, of course)

[dangotbanned]

pandas has something similar (import_optional_dependency in their source code) which we could probably use (with attribution, of course)

Here's the polars one - which is pretty clean and well-documented

https://github.com/pola-rs/polars/blob/0ca485a88a217eab2d55efbd981dfd68709e63be/py-polars/polars/dependencies.py#L225-L287

[dangotbanned]

AFAIK, if we want the imported modules to be understood by a type checker - we'll need 1 function per-import.

Example

_module_not_found

from __future__ import annotations

from importlib import import_module
from importlib.util import find_spec

from types import ModuleType
from typing import Any
from typing import Literal

def _module_not_found(module_name: str, **kwds: Any) -> ModuleNotFoundError:
    some_logic_transforming_kwds = repr(kwds)
    msg = f"{module_name!r} not found.\n{some_logic_transforming_kwds!r}"
    return ModuleNotFoundError(msg)

importlib-only

By default, we know nothing about the the contents of the module.

def import_optional_1(module_name: Literal["polars", "pandas"], **kwds: Any):
    if find_spec(module_name):
        return import_module(module_name)
    raise _module_not_found(module_name=module_name, **kwds)

Single function

Better than the first option.
But we can't capture @overload(s)

def import_optional_2(module_name: Literal["polars", "pandas"], **kwds: Any):
    if module_name == "polars" and find_spec("polars"):
        import polars as pl

        return pl
    elif module_name == "pandas" and find_spec("pandas"):
        import pandas as pd

        return pd

    raise _module_not_found(module_name=module_name, **kwds)

Multiple functions

So far, this is the only route I've found that would preserve typing

def import_polars(**kwds: Any):
    if find_spec("polars"):
        import polars as pl

        return pl

    raise _module_not_found(module_name="polars", **kwds)


def import_pandas(**kwds: Any):
    if find_spec("pandas"):
        import pandas as pd

        return pd

    raise _module_not_found(module_name="pandas", **kwds)

Important

The warning above is something we don't want to lose

[EdAbati]

Other examples (in case we need more :) )

• https://pypi.org/project/lazy-imports/
• optuna try_import