[BLOG] Update on Array API adoption in scikit-learn#946
[BLOG] Update on Array API adoption in scikit-learn#946rgommers merged 14 commits intoQuansight:mainfrom
Conversation
lucyleeow
requested review from
pavithraes,
rgommers and
trallard
as code owners
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
| ### Vendoring `array-api-compat` and `array-api-extra` | ||
|
|
||
| Scikit-learn now vendors both [`array-api-compat`](https://data-apis.org/array-api-compat/) and [`array-api-extra`](https://data-apis.org/array-api-extra/). `array-api-compat` is a wrapper around common array libraries (e.g., PyTorch, CuPy, JAX) that bridges gaps to ensure compatibility with the standard. It enables adoption of backwards incompatible changes while still allowing array libraries time to adopt the standard slowly. `array-api-extra` provides array functions not included in the standard but deemed useful for array-consuming libraries. | ||
|
|
||
| We chose to vendor these now much more mature libraries, to avoid the complexity of conditionally handling optional dependencies throughout the codebase. This approach also follows precedent, as also SciPy vendors these packages. |
There was a problem hiding this comment.
out of interest, have there been any downsides to vendoring?
There was a problem hiding this comment.
I'm afraid I was not around for the full discussion but Olivier summarises the pros and cons quite well here:
scikit-learn/scikit-learn#30340 (comment)
There was also some discussion about workflow around getting new functions into array-api-extra - which is handled as before, write private functions and every now and again add them to array-api-extra, so there is no need to wait for array-api-extra to get access to the new functions (scikit-learn/scikit-learn#30340 (comment))
betatim
left a comment
betatim
left a comment
There was a problem hiding this comment.
Nice blog post!
I left a few comments.
|
|
||
| The array API standard adopted DLPack as the recommended [data interchange](https://data-apis.org/array-api/latest/design_topics/data_interchange.html#data-interchange) protocol. This protocol is widely implemented in array libraries and offers an efficient, C ABI compatible protocol for array conversion. While this provided us with an easy way to implement these transfers, there were limitations. Cross-device transfer capability was only introduced in DLPack v1, released in September 2024. This meant that only the latest PyTorch and CuPy versions have support for DLPack v1. Moreover, not all array libraries have adopted support yet. We therefore implemented a 'manual' fallback though this requires conversion via NumPy when the transfer involves two non-NumPy arrays. Additionally, there are no DLPack tests in [array-api-tests](https://github.com/data-apis/array-api-tests), a testing suite to verify standard compliance, leaving DLPack implementation bugs easier to overlook. Despite these challenges, scikit-learn will benefit from future improvements, such as addition of a C-level API for DLPack exchange that bypasses Python function calls, offering significant benefit for GPU applications. | ||
|
|
||
| Beyond the technical considerations, there were also user interface considerations. How should we inform users that these conversions, which incur memory and performance cost, are occurring? We decided against warnings, which risk being ignored or becoming a nuisance, and instead clearly document this behaviour. Further, we need to determine how to gracefully handle device-specific data type limitations, specifically MPS does not support float64. This requires downcasting which must be clearly communicated to users. |
There was a problem hiding this comment.
The comment about MPS at the end is interesting, but feels like it got tacked on to a paragraph that is vaguely related. It comes without warning and then ends as quickly as it started. Not sure if it is worth making a whole paragraph about this, if not maybe it is also not worth having it at all? Or have a two-three sentence paragraph about this to explain that different devices have different precision and that how to handle this requires a bit of thought?
There was a problem hiding this comment.
Good point and honestly the MPS downcasting this is probably still a WIP in how it's handled. I think I will do this
or have a two-three sentence paragraph about this to explain that different devices have different precision and that how to handle this requires a bit of thought?
since this is not completely resolved yet 😬
ogrisel
left a comment
ogrisel
left a comment
There was a problem hiding this comment.
I did another pass and this reads great to me. Thanks @lucyleeow!
StefanieSenger
left a comment
StefanieSenger
left a comment
There was a problem hiding this comment.
Great article, @lucyleeow!
I've found a few typo/wording issues (see comments).
apps/labs/public/posts/array-api-scikit-learn-2026/array-api-scikit-learn-2026-feature.png
Show resolved
Hide resolved
rgommers
left a comment
rgommers
left a comment
There was a problem hiding this comment.
Looks great, thank you @lucyleeow and all reviewers!
Co-authored-by: Pavithra Eswaramoorthy <pavithraes@outlook.com>
|
Thanks all! 🙏 |
8 participants
Text styling
Non-text contents