test: ideas to speed up test suite

[MarcoGorelli]

I don't know what the solution is, but the test suite is getting painfully slow

Any investigations into what the bottlenecks are, and how we might go about speeding it up, would be welcome 🙏

[MarcoGorelli]

running

uv pip install -e ".[dask, modin, ibis]" --group core-tests --group extra -U
pytest tests --cov=narwhals --cov=tests --cov-fail-under=100 --runslow --constructors=pandas,pandas[nullable],pandas[pyarrow],pyarrow,modin[pyarrow],polars[eager],polars[lazy],dask,duckdb,sqlframe,ibis --durations=30

locally i see

4.85s call     tests/hypothesis/join_test.py::test_left_join
3.86s call     tests/frame/invalid_test.py::test_memmap
3.70s call     tests/expr_and_series/rolling_sum_test.py::test_rolling_sum_hypothesis
3.39s call     tests/expr_and_series/cast_test.py::test_cast_series[modin[pyarrow]]
3.17s call     tests/hypothesis/join_test.py::test_join
2.96s call     tests/expr_and_series/cast_test.py::test_cast[modin[pyarrow]]
2.70s call     tests/expr_and_series/binary_test.py::test_expr_binary[dask]
2.62s call     tests/joblib_test.py::test_parallelisability[pyarrow]
2.61s call     tests/v1_test.py::test_toplevel[modin[pyarrow]]
2.57s call     tests/expr_and_series/binary_test.py::test_expr_binary[modin[pyarrow]]
2.35s call     tests/hypothesis/join_test.py::test_cross_join
2.34s call     tests/series_only/hist_test.py::test_hist_bin_hypotheis[pandas]
2.34s call     tests/expr_and_series/horizontal_broadcasts_test.py::test_sumh_broadcasting[modin[pyarrow]]
2.31s call     tests/expr_and_series/concat_str_test.py::test_concat_str[modin[pyarrow]-True-expected0]
2.07s call     tests/joblib_test.py::test_parallelisability_series[modin[pyarrow]]
1.99s call     tests/joblib_test.py::test_parallelisability[polars[eager]]
1.91s call     tests/expr_and_series/dt/total_minutes_test.py::test_total_minutes
1.81s call     tests/joblib_test.py::test_parallelisability[modin[pyarrow]]
1.80s call     tests/expr_and_series/all_horizontal_test.py::test_allh_iterator[modin[pyarrow]]
1.79s call     tests/series_only/hist_test.py::test_hist_count_hypothesis[pandas]
1.67s call     tests/expr_and_series/concat_str_test.py::test_concat_str[modin[pyarrow]-False-expected1]
1.56s call     tests/hypothesis/getitem_test.py::test_getitem[pyarrow_table_constructor]
1.52s call     tests/expr_and_series/unary_test.py::test_unary[modin[pyarrow]]
1.49s call     tests/series_only/hist_test.py::test_hist_bin_hypotheis[pyarrow]
1.47s call     tests/expr_and_series/fill_null_test.py::test_fill_null_limits[modin[pyarrow]]
1.47s call     tests/expr_and_series/rolling_mean_test.py::test_rolling_mean_hypothesis
1.45s call     tests/frame/group_by_test.py::test_group_by_expr[modin[pyarrow]-keys5-aggs5-expected5-sort_by5]
1.44s call     tests/expr_and_series/reduction_test.py::test_empty_scalar_reduction_with_columns[modin[pyarrow]]
1.40s call     tests/expr_and_series/dt/datetime_duration_test.py::test_duration_attributes_series[modin[pyarrow]-total_seconds-expected_a1-expected_b1-expected_c1]
1.34s call     tests/expr_and_series/rolling_var_test.py::test_rolling_var_hypothesis

so, uninstall modin when running tests locally i guess

[dangotbanned]

I need to fix my local modin install, but I recall this having a speedup when I had it working.

Possibly zero-copy related?

diff --git a/tests/conftest.py b/tests/conftest.py
index 26c8f114a..0153a2e7b 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -110,7 +110,7 @@ def modin_pyarrow_constructor(obj: Data) -> NativeFrame:  # pragma: no cover
     import modin.pandas as mpd
     import pandas as pd
 
-    df = mpd.DataFrame(pd.DataFrame(obj)).convert_dtypes(dtype_backend="pyarrow")
+    df = mpd.DataFrame(pd.DataFrame(obj).convert_dtypes(dtype_backend="pyarrow"))
     return cast("NativeFrame", df)

Maybe worth trying to optimize all the constructor functions?

Even if they aren't slow individually, there's so many calls to them that I imagine it adds up

Update

Important

We should document that modin requires an environment variable set
MODIN_ENGINE=python
We have that for pytest, but not anywhere else

[dangotbanned]

We could also try out pytest-xdist, that had a pretty huge impact for us in altair (even in CI)

• refactor: Add ruff rules, improve type annotations, improve ci performance vega/altair#3431 (comment)
• refactor: Add ruff rules, improve type annotations, improve ci performance vega/altair#3431 (comment)

The only caveat is some tests might not work out-of-the-box, if they mutate any global state

[MarcoGorelli]

thanks - sure might help to try it (though i suspect we may end up fork bombing ourselves)

[FBruzzesi]

Github-hosted runners updated to 4 cores for public repos last year: see Standard GitHub-hosted runners for public repositories.

Locally I am getting a good 3x speedup by using pytest-xdist and -n 4 flag

[FBruzzesi]

Another idea is to have a few dataframes that can be used without having to call constructors each test. That would also help to understand when we return a copy vs modify the original object.

This might be a huge refactor of the test suite though :)

[dangotbanned]

Even though they're disabled for most CI runs, we really should investigate how to speed up the pyspark tests.

I feel like it must be either a config or implementation issue - with how much slower they are than every other backend.

Haven't had much luck getting pyspark running locally, but if anyone's interested ... 😂

[dangotbanned]

This wouldn't impact local performance, but maybe we could review if there are any redundancies in the workflows?

• https://github.com/narwhals-dev/narwhals/tree/81edd31770b84d4dafd50276673affc099527ac4/.github/workflows

During (#2679), I must say I wasn't that confident I understood all the various configs/version ranges being tested

[dangotbanned]

Another idea is to have a few dataframes that can be used without having to call constructors each test. That would also help to understand when we return a copy vs modify the original object.

This might be a huge refactor of the test suite though :)

I think that is something you can control with pytest fixtures isn't it?

Possibly the scope?

[dangotbanned]

• test: ideas to speed up test suite #2818 (comment)
• test: ideas to speed up test suite #2818 (comment)

Ignore these ^

I botched the issue number 🤦‍♂️