[Enh]: Allow `LazyFrame.with_row_index(..., order_by=...)`

[FBruzzesi]

Description

Proposing to allow the following: LazyFrame.with_row_index(name, order_by=into_exprs)

This would be a deviation from polars itself.
In a way this is similar to #2174

Suggested implementation

I am missing some detail for sure, but it would be something close to:

self.with_column(**{name: nw.lit(1).cum_sum().over(order_by=order_by) - 1})

[dangotbanned]

@FBruzzesi this is news to me, seems like polars added LazyFrame.with_row_index around 0.20.*

• https://docs.pola.rs/api/python/stable/reference/lazyframe/api/polars.LazyFrame.with_row_index.html

I've only found it mentioned in relation to the new streaming engine in issues.

The last time I tried mixing an index with anything LazyFrame, I always found it much more reliable to just do:

LazyFrame.collect().with_row_index().lazy()

However things may have improved and it there might be stronger ordering guarantees

[MarcoGorelli]

out of interest, what's the use-case?

i'd be more inclined to deprecate this for lazyframe, just having it around feels like it's got the potential to be misued

[dangotbanned]

out of interest, what's the use-case?

+1

i'd be more inclined to deprecate this for lazyframe, just having it around feels like it's got the potential to be misued

Huh, I didn't know we supported it yet for nw.LazyFrame 🤔
I agree though on the potential for misuse

[FBruzzesi]

out of interest, what's the use-case?

I sporadically check if some code I have for personal or work use can be narwhalified. This is one of the feature that we have but basically it's supported only by polars (or eager frame casted to lazy)

just having it around feels like it's got the potential to be misued

Not sure why that would be different than methods that require .over(order_by). But yes I would rather deprecate it than keeping it for polars only

[MarcoGorelli]

Thinking about this further, I think the "subset of Polars" way to do this would be to introduce nw.int_range, and then use that with over

So my suggestion would be:

• deprecate LazyFrame.with_row_index
• introduce nw.int_range, which is a window function, meaning that for LazyFrame it needs to be followed by over(..., order_by=...)

Does this sound good? If so, @FBruzzesi did it interest you to try this out?

[dangotbanned]

I've been waiting for someone to suggest supporting any of the pl.*_range functions 🎉

@MarcoGorelli could you burst my bubble and let me know that only nw.int_range is a realistic option?

[MarcoGorelli]

😄 which one(s) did you want to avoid?

[dangotbanned]

😄 which one(s) did you want to avoid?

Avoid? Did you mean add?

I'd be interested in the temporal ones, like pl.date_range - since we'd need to start thinking more about duration string language
Realistically though, those worms may be best left canned 😉

[MarcoGorelli]

that's a good one

i'd like to avoid the horizontal ones for now (int_ranges, date_ranges, ...)

[FBruzzesi]

Does this sound good? If so, @FBruzzesi did it interest you to try this out?

Yes it definitly does! I will try to tackle it - if you don't see anything by end of week it might mean I had no much time 😅

[FBruzzesi]

Update: while this is straightforward for eager backends, for lazy ones int_range is fairly more complex than a simple row_index. However I don't really see a point in supporting int_range for eager backends only since this issue started with the request of supporting something for lazy ones 🥲

Based on the following statement:

if you don't see anything by end of week it might mean I had no much time 😅

you can now assume I didn't finish this in time 😂