[Enh]: Add `pivot` to Arrow

[tylerriccio33]

We would like to learn about your use case. For example, if this feature is needed to adopt Narwhals in an open source project, could you please enter the link to it below?

No response

Please describe the purpose of the new feature or describe the problem to solve.

Implement the pivot function for Arrow. Discovered as missing in #2177.

Suggest a solution if possible.

No response

If you have tried alternatives, please describe them below.

No response

Additional information that may help us understand your needs.

No response

[MarcoGorelli]

thanks - this may need implementing a bit manually as in the last example from https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.pivot.html

[dangotbanned]

Somewhat recently, pyarrow added (pa.compute.pivot_wider) that might help.

Pretty classic pyarrow to manage to not add this as a method on pa.Table 🙃

[FBruzzesi]

Added high prio label as it was requested in discord as well - I think it is both in scope and doable (must double check performance though 🙈)

[dangotbanned]

I'm probably going to need to implement this in #2572, for the same reasons as #3364

If anyone's started/starting work on this - let me know!
It's lower down on my list than the others, still need to figure out how/if pyarrow.compute.pivot_wider can be used for this.

Related tests

• Scalar aggregate (python)
• Hash aggregate (c++)

[dangotbanned]

I'm probably going to need to implement this in #2572, for the same reasons as #3364

Just to update anyone following this ...

I'm fairly happy with where #3373 is at now and can confirm DataFrame.pivot is possible in pyarrow.

I've isolated most of it to a single module narwhals/_plan/arrow/pivot.py, which should help when adapting for main.

I also went digging for some extra tests from polars and discovered a bug that I've reported in (apache/arrow#48679).
That issue has an example of how to handle most of the simplest case (on: str, index: str, values: str) natively.

---

The main problems to solve, in order of difficulty, are:

1. Multiple values columns
2. Multiple on columns
3. Generating the correct output column names
4. Supporting aggregate_function