[BLOG]: Supporting dask arrays in scipy via the Python Array API standard

[lithomas1]

Text styling

• The blog is written with plain language (where relevant).
• If there are headers, they use the proper header tags in order to do so (with only one level-one header).
• All links describe where they link to (for example, check the Quansight labs website).
• Any kind of styling that the author uses (for example, bold for emphasis) is consistent throughout the blog.

Non-text contents

• Blog post featured image is in PNG or JPEG format, not SVG.
• All content is represented as text (for example, images need alt text and videos need captions or descriptive transcripts).
• If there are emojis, there are not more than three in a row.
• Don't use flashing gifs or videos.
• If it were to be read as plain text, the blog still makes sense and no information is missing.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
labs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 6, 2025 0:31am

[lithomas1]

Not done yet, but just wanted to put up a quick draft of my blog post.

The main omission is probably the case study section where I port a scipy.stats workflow to dask arrays and look at performance.

All other sections are basically complete.

[rgommers]

This looks like a good start @lithomas1, thanks. The flow of the story at a high level looks good to me. The case study section is important indeed; the post is still pretty draft now so I didn't review in detail.

[lithomas1]

@rgommers This should be ready for a look now.

[pavithraes]

@lithomas1 Thank you! This is a good read!

I've shared some suggestions, mainly around phrasing :)

[pavithraes]

Just noting that we'll need to update this before merging. :)

[pavithraes]

Suggested change

	In this post, I describe my journey getting SciPy to work with Dask arrays natively via the array API and the current
	limitations and future outlook.
	In this post, I describe my journey getting SciPy to work with Dask Arrays natively via the Array API standard,
	and discuss the current limitations and future outlook of this work.

[pavithraes]

Suggested change

	For those unfamiliar, the [Python Array API standard](https://data-apis.org/array-api/latest/API_specification/),
	The [Python Array API standard](https://data-apis.org/array-api/latest/API_specification/),

I think it'll be nicer to start with a simpler statement

[pavithraes]

Suggested change

	is a specification aimed at unifying the various APIs of different array libraries (e.g. Numpy, PyTorch, JAX, Dask, etc.).
	is a specification aimed at unifying the various APIs of different array libraries (e.g. NumPy, PyTorch, JAX, Dask, etc.).

I see there are several different capitalizations for various libraries through the blog. Could you please do a find and replace to have one style for all?

I think these are the capitalizations: NumPy, SciPy, Dask, PyTorch, pandas, JAX, and CuPy, unless you're referring to the API, in which case it's all lowercase and presented as inline code like dask.array. I think this is already the case for the most part, but I noticed a few deviations here and there, hence the explicit comment. :)

[pavithraes]

Suggested change

	way since mid 2023 when array API support was first experimentally introduced within the libary. While the array API
	way since mid-2023 when array API support was first experimentally introduced within the library. While the array API

[pavithraes]

Suggested change

	See [here](https://scipy.github.io/devdocs/dev/api-dev/array_api.html#currently-supported-functionality)
	See the [SciPy developer docs](https://scipy.github.io/devdocs/dev/api-dev/array_api.html#currently-supported-functionality)

[pavithraes]

Suggested change

	As of today, the `scipy.fft/special/stats` modules have the best support for dask arrays today, and are able to
	The `scipy.fft/special/stats` modules have the best support for dask arrays today, and are able to

[pavithraes]

Really like this!

[pavithraes]

Suggested change

While we weren't entirely able to avoid computation in the middle (dask still struggles with unknown shapes which we get through our boolean masking on the dataframe), we were able to entirely keep the computation in dask. This is a big improvement over the pre-Array API behavior where the input dask arrays would be cast to numpy arrays (forcing computation and storage of intermediate results in one worker which can lead to performance degredation and out-of-memory errors)

While we weren't entirely able to avoid computation in the middle (dask still struggles with unknown shapes which we get through our boolean masking on the dataframe), we were able to entirely keep the computation in dask. This is a big improvement over the pre-Array API behavior where the input dask arrays would be cast to numpy arrays (forcing computation and storage of intermediate results in one worker which can lead to performance degredation and out-of-memory errors).

[pavithraes]

Suggested change

	[attempt](https://github.com/scikit-learn/scikit-learn/pull/28588) to add array API support within scikit-learn stalled
	[attempt (scikit-learn PR#28588)](https://github.com/scikit-learn/scikit-learn/pull/28588) to add array API support within scikit-learn stalled

[lysnikolaou]

@rgommers @pavithraes Do you think it's worth it for me to address the feedback comments on this and try to get it merged or should we close it?

[rgommers]

I think it's close to done, and would be nice to publish still as it captures interesting work done under a grant.

@lithomas1 I think you planned to revisit this still, and it fell through the cracks? Any thoughts here?