refactor: Add `CompliantDataFrame.from_arrow` by dangotbanned · Pull Request #2306 · narwhals-dev/narwhals

dangotbanned · 2025-03-28T12:51:59Z

What type of PR is this? (check all applicable)

Related issues

Part of API: io functions for v2 #2116

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

If you have comments or can explain your changes, please do so below

Refine the error handling for each backend
- Most of it can live in ArrowDataFrame.from_arrow
Widen to accept pa.Table
- since we support pyarrow>=11, I think this function can still provide some value for those earlier versions
- The Arrow* impl has similar logic to pa.table in pyarrow==13.0.0

- Defined `ArrowStreamExportable` at runtime here - Currently, just a typing-only def in `nw.functions`

Marked as draft since I'll do another refactor pass

Couldn't understand the `TypeVar` that `_PandasConvertible` has resolved when partially binding the method

Makes more sense for the logic to stay there

- Still need to refine the error handling for each backend - Most of it can live in `ArrowDataFrame.from_arrow` - Also want to widen this to accept `pa.Table` - since we support `pyarrow>=11`, I think this function still provide some value for those earlier versions

https://results.pre-commit.ci/run/github/760058710/1743166321.1j7RUHCQRaOqsEVGmKBjBw

Helper that `polars` and `pandas` can share, without needing to worry about missing imports

- First is for non-`pa.Table` before `14` - Second is just for exhaustiveness

I only needed it when using a callable, but removed that route

https://results.pre-commit.ci/run/github/760058710/1743170690.NGBs6mxrTqGQAiGuhrhWqQ

I only wanted a one line description 😞 https://results.pre-commit.ci/run/github/760058710/1743170690.NGBs6mxrTqGQAiGuhrhWqQ

https://github.com/narwhals-dev/narwhals/actions/runs/14131536804/job/39592845144?pr=2306

dangotbanned · 2025-03-28T15:40:44Z

@MarcoGorelli I'm close to finishing, just wanted to check in and see if you had any thoughts on supporting pa.Table here?

I left some notes in the description on why it makes sense to me 🙂

Would be very easy to support across all versions

MarcoGorelli · 2025-03-28T16:15:15Z

thanks @dangotbanned ! sure, should be good to accept pa.table pre-pycapsule

dangotbanned · 2025-03-28T16:43:46Z

sure, should be good to accept pa.table pre-pycapsule

Perfect, thanks @MarcoGorelli

Still need to test how pa.table responds to pa.Table for post-pycapsule.
I think it would use __arrow_c_stream__ but there's only one way to find out

Update

Yeah, all good!

`pyarrow>=14`

import pyarrow as pa

data = {"ab": [1, 2, 3], "ba": [4, 5, 6]}
df = pa.table(data)
>>> df
pyarrow.Table
ab: int64
ba: int64
----
ab: [[1,2,3]]
ba: [[4,5,6]]

repeat = pa.table(df)
>>> repeat
pyarrow.Table
ab: int64
ba: int64
----
ab: [[1,2,3]]
ba: [[4,5,6]]

So we can just document support for both - and what I have already adds support for pa.Table w/ pyarrow<14:

IntoArrowTable: TypeAlias = ArrowStreamExportable | pa.Table

Resolves #2306 (comment)

- Following the new changes, this path should work (adb6bf1) - Unclear on whether the `pandas` import will occur though

dangotbanned · 2025-03-28T18:09:08Z

tests/from_pycapsule_test.py

    monkeypatch.delitem(sys.modules, "pandas")
-    df = nw.from_native(tbl, eager_only=True)
-    result = nw.from_arrow(df, backend=pl)
+    if PYARROW_VERSION < (14,):


https://github.com/narwhals-dev/narwhals/actions/runs/14134927029/job/39604239875?pr=2306

Feel like this would be good to have coverage for, but not sure how feasible that is with the current CI.
Adding a # pragma: no cover seems wrong

Also, it means the previous xfail(s) were never triggered?

@MarcoGorelli I was planning to adapt the rest of the tests in a similar way following

adb6bf1

Do you have a preference on what to do here?

Added a no cover for now in (7bf8b27)

no strong preference

Pending #2306 (comment)

dangotbanned · 2025-03-28T18:43:26Z

narwhals/_arrow/dataframe.py

+        backend_version = context._backend_version
+        if isinstance(data, pa.Table):
+            native = data
+        elif backend_version >= (14,) or isinstance(data, Collection):


Supporting Collection here might seem strange.

My thinking is that were a list | tuple | dict of pyarrow objects passed - pa.table would still be capable of turning that into a pa.Table:
https://github.com/apache/arrow/blob/7c1b96e26910dd553ee0ccf0792e1854af6b0021/python/pyarrow/table.pxi#L4993-L5000

MarcoGorelli

nice, thanks @dangotbanned !

MarcoGorelli · 2025-03-29T21:59:17Z

narwhals/_pandas_like/dataframe.py

+            msg = "congratulations, you entered unreachable code - please report a bug"
+            raise AssertionError(msg)


lol, can't believe i wrote this, we may want to make a utility with a consistent message for these at some point

Certainly made me smile when I copied it over 😄

dangotbanned added 10 commits March 28, 2025 11:05

feat(typing): Add ArrowConvertible

12330d5

- Defined `ArrowStreamExportable` at runtime here - Currently, just a typing-only def in `nw.functions`

feat: Add CompliantDataFrame.from_arrow

83b2752

feat(DRAFT): Add PolarsDataFrame.from_arrow

7c306dc

Marked as draft since I'll do another refactor pass

feat(DRAFT): Add ArrowDataFrame.from_arrow

038c8da

feat(DRAFT): Add PandasLikeDataFrame.from_arrow

51bef83

chore: Add version to _from_arrow_impl

79f080a

fix(typing): Help mypy

1812d28

Couldn't understand the `TypeVar` that `_PandasConvertible` has resolved when partially binding the method

refactor: Reuse ArrowDataFrame impl for PolarsDataFrame

1f92a58

Makes more sense for the logic to stay there

fix: UnboundLocalError oops

41e4034

dangotbanned added internal pyarrow Issue is related to pyarrow backend labels Mar 28, 2025

dangotbanned mentioned this pull request Mar 28, 2025

API: io functions for v2 #2116

Open

6 tasks

dangotbanned added 10 commits March 28, 2025 12:54

ignore banned import

8d95cb2

https://results.pre-commit.ci/run/github/760058710/1743166321.1j7RUHCQRaOqsEVGmKBjBw

recover PandasLike assertion error

299be9c

fix: correct typo in original message

976cffb

chore: Add _into_arrow_table

7a31e65

Helper that `polars` and `pandas` can share, without needing to worry about missing imports

refactor: Use _into_arrow_table

f5c79cc

chore: Fill in missing error branches

e84a035

- First is for non-`pa.Table` before `14` - Second is just for exhaustiveness

chore(typing): Remove unused native annotation

392a611

I only needed it when using a callable, but removed that route

fix typo

cb6c86f

https://results.pre-commit.ci/run/github/760058710/1743170690.NGBs6mxrTqGQAiGuhrhWqQ

docs: Satisfy darglint

883b927

I only wanted a one line description 😞 https://results.pre-commit.ci/run/github/760058710/1743170690.NGBs6mxrTqGQAiGuhrhWqQ

test: Get and ignore more coverage

655da1d

https://github.com/narwhals-dev/narwhals/actions/runs/14131536804/job/39592845144?pr=2306

dangotbanned added 2 commits March 28, 2025 17:36

docs: Add IntoArrowTable alias

adb6bf1

Resolves #2306 (comment)

test: Try test_from_arrow_to_polars

bdeaaac

- Following the new changes, this path should work (adb6bf1) - Unclear on whether the `pandas` import will occur though

dangotbanned commented Mar 28, 2025

View reviewed changes

dangotbanned changed the title ~~refactor(DRAFT): Add CompliantDataFrame.from_arrow~~ refactor: Add CompliantDataFrame.from_arrow Mar 28, 2025

test: no cover

7bf8b27

Pending #2306 (comment)

dangotbanned commented Mar 28, 2025

View reviewed changes

dangotbanned marked this pull request as ready for review March 28, 2025 18:43

dangotbanned added 2 commits March 28, 2025 22:49

Merge branch 'main' into df-from-arrow

3843bc2

Merge branch 'main' into df-from-arrow

ced6d07

dangotbanned mentioned this pull request Mar 29, 2025

chore: Tidy up PolarsExpr #2313

Merged

10 tasks

Merge branch 'main' into df-from-arrow

5802361

MarcoGorelli approved these changes Mar 29, 2025

View reviewed changes

dangotbanned merged commit a9e06d9 into main Mar 29, 2025
28 checks passed

dangotbanned deleted the df-from-arrow branch March 29, 2025 22:29

dangotbanned mentioned this pull request Apr 4, 2025

enh: Add Compliant.from_* constructors #2342

Closed

5 tasks

dangotbanned mentioned this pull request Jan 29, 2026

[Enh]: add LazyFrame.collect_batches #3360

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: Add `CompliantDataFrame.from_arrow`#2306

refactor: Add `CompliantDataFrame.from_arrow`#2306
dangotbanned merged 26 commits intomainfrom
df-from-arrow

dangotbanned commented Mar 28, 2025 •

edited

Loading

Uh oh!

dangotbanned commented Mar 28, 2025

Uh oh!

MarcoGorelli commented Mar 28, 2025

Uh oh!

dangotbanned commented Mar 28, 2025 •

edited

Loading

Uh oh!

dangotbanned Mar 28, 2025

Uh oh!

dangotbanned Mar 28, 2025

Uh oh!

dangotbanned Mar 28, 2025

Uh oh!

dangotbanned Mar 28, 2025

Uh oh!

MarcoGorelli Mar 29, 2025

Uh oh!

dangotbanned Mar 28, 2025

Uh oh!

MarcoGorelli left a comment

Uh oh!

MarcoGorelli Mar 29, 2025

Uh oh!

dangotbanned Mar 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		msg = "congratulations, you entered unreachable code - please report a bug"
		raise AssertionError(msg)

Conversation

dangotbanned commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

Uh oh!

dangotbanned commented Mar 28, 2025

Uh oh!

MarcoGorelli commented Mar 28, 2025

Uh oh!

dangotbanned commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update

pyarrow>=14

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MarcoGorelli left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dangotbanned commented Mar 28, 2025 •

edited

Loading

dangotbanned commented Mar 28, 2025 •

edited

Loading

`pyarrow>=14`