enh: Passing arguments to `pa.Table.to_pandas`

[MarcoGorelli]

Discussed in #2121

^{Originally posted by dhirschfeld February 28, 2025}
In my current code I have a very large pa.Table that I want to convert to pd.DataFrame. In the to_pandas call I specify split_blocks and self_destruct as I don't want 2 copies of the data in memory.

I'm thinking of porting this code to narwhals so was wondering if there's a way to do the same - convert to a pa.Table to different format without doubling the memory requirements?

I'm interest in converting to either pandas or polars hence why I thought narwhals might be a good fit.

[dangotbanned]

I think improved performance characteristics are a good reason to not unify things like (#2056 (comment)).

If anyone wants to pursue this - I've added a branch with the typing tools to support it main...pa-table-to-pandas

narwhals/narwhals/_arrow/typing.py

Lines 61 to 79 in 112cfd9

	class ArrowTableToPandasKwds(TypedDict, total=False):
	"""Keyword arguments for [pyarrow.Table.to_pandas](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pandas)."""
	memory_pool: pa.MemoryPool | None
	categories: list[Any] | None
	strings_to_categorical: bool
	zero_copy_only: bool
	integer_object_nulls: bool
	date_as_object: bool
	timestamp_as_object: bool
	use_threads: bool
	deduplicate_objects: bool
	ignore_metadata: bool
	safe: bool
	split_blocks: bool
	self_destruct: bool
	maps_as_pydicts: Literal["None", "lossy", "strict"] | None
	types_mapper: Callable[[pa.DataType], PandasExtensionDtype | None] | None
	coerce_temporal_nanoseconds: bool

You'd still need to add the runtime handling of providing the wrong keywords.
Statically - they'd only appear for pa.Table:

narwhals/narwhals/dataframe.py

Lines 658 to 665 in 112cfd9

	@overload
	def to_pandas(
	self: DataFrame[pa.Table], **kwds: Unpack[ArrowTableToPandasKwds]
	) -> pd.DataFrame: ...
	@overload
	def to_pandas(self: Self, **kwds: Any) -> pd.DataFrame: ...
	def to_pandas(self: Self, **kwds: Any) -> pd.DataFrame:
	"""Convert this DataFrame to a pandas DataFrame.

[dangotbanned]

Coming back to this as we missed something that I noticed in #2879

The request said:

In my current code I have a very large pa.Table that I want to convert to pd.DataFrame. In the to_pandas call I specify split_blocks and self_destruct as I don't want 2 copies of the data in memory.

But I wasn't aware at the time that this was something you can also do in polars:

• https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.to_pandas.html
• https://docs.pola.rs/api/python/stable/reference/series/api/polars.Series.to_pandas.html

I want to add this for preserving nested data types in pandas, when moving back and forth with pyarrow

The first use-case I have in mind is (#1561), which we can do pretty easily it turns out 😄

• refactor: Simplify compliant Series.hist #2839 (comment)
• refactor: Simplify compliant Series.hist #2839 (comment)

For pandas, we could default to deferring to pyarrow when working on anything .struct or .list related (given that it is a dependency)

These are all the other issues I've found related to nested data types:

• [Enh]: Add list.get() negative indices and out-of-bounds access #2905
• enh: add implode to allow nw.group_by('a').agg(nw.col('b').implode()) (similar to polars pl.group_by('a').agg(pl.col('b'))) #2660
• [Enh]: Support int_range #2722