api: use `large_string` / `large_list` when converting Narwhals types -> PyArrow?

[MarcoGorelli]

Currently we convert nw.String to pyarrow string. Polars would convert pl.String to pyarrow large_string. We should probably be doing the same?

[dangotbanned]

Sounds the right move to me.

@MarcoGorelli maybe you'd be able to reveal a bug in the current implementation - by passing a string that would trigger an overflow?

I've no idea if we'd have coverage for that already in a test

[MarcoGorelli]

I think we should treat this as a bug

df = pa.table({
    'store': ['foo', 'bar'],
    'item': ['axe', 'saw']
}, schema=pa.schema([('store', pa.large_string()), ('item', pa.large_string())]))

print(nw.from_native(df).with_columns(store_item = nw.concat_str('store', 'item', separator='-')))

results in

┌───────────────────────────────────┐
|        Narwhals DataFrame         |
|-----------------------------------|
|pyarrow.Table                      |
|store: large_string                |
|item: large_string                 |
|store_item: string                 |
|----                               |
|store: [["foo","bar"]]             |
|item: [["axe","saw"]]              |
|store_item: [["foo-axe","bar-saw"]]|
└───────────────────────────────────┘

So, two large_strings got concatenated into a string. I'd have expected them to get concatenated into a large-string

[dangotbanned]

Does pyarrow keep the result as large_string if you rewrite (#2097 (comment)) using only native functions?

Seems plausible that it could be down-casting to reduce memory usage if it wasn't needed?

Do agree it is a bug if we aren't matching the native behavior though

[MarcoGorelli]

Does pyarrow keep the result as large_string if you rewrite (#2097 (comment)) using only native functions?

yup

[MarcoGorelli]

In [5]: pc.binary_join_element_wise(df['store'], df['item'])
Out[5]: 
<pyarrow.lib.ChunkedArray object at 0x7f09e680d8a0>
[
  [
    "foo",
    "bar"
  ]
]

In [6]: pc.binary_join_element_wise(df['store'], df['item']).type
Out[6]: DataType(large_string)

[dangotbanned]

Does pyarrow keep the result as large_string if you rewrite (#2097 (comment)) using only native functions?

yup

🐛 🐛 🐛

[MarcoGorelli]

The "concatenating strings" part is definitely a bug, and we'll fix it for the next release, but I'm less sure about mapping large_string to string

First, because mixing them doesn't seem very well-supported in PyArrow apache/arrow#45717

Second, because PyArrow defaults to string for pa.scalar('a'), and in read_csv. So, we should probably respect their default