Align conda and wheel building workflows

[vyasr]

Historically our conda and wheel GHA workflow scripts have looked fairly different for a number of reasons. However, with #33 many of the fundamental distinctions will no longer exist because wheels will also have separate build steps for C++ and Python builds. As a result, we should invest in aligning our workflows as much as possible so as to reduce maintenance costs going forward. Some changes that we ought to make:

• We should parallelize test jobs across different Python packages. Wheels generally already do this, while all conda tests typically occur within a single job (in some repos there is a partial split, usually based on criteria specific to each repo e.g. cuml dask tests or cudf tests that aren't part of the cudf package).
• We should standardize handling of pure Python packages. Wheels already do this, while conda packages do not. See Properly support building pure Python packages #43 for a more detailed writeup.
• We should automatically append the CUDA version to the artifacts produced by the gha-tools for uploading packages. We already do this for conda, but not for wheels. As part of Support dynamic linking between RAPIDS wheels #33 we will be rearchitecting the wheel pipelines to have one job for building the C++ wheel and one for all the Python wheels, matching the conda packages more closely (this becomes feasible after Support dynamic linking between RAPIDS wheels #33 because the Python builds will be very fast if they don't have to rebuild the C++). Once that is the case, we can also upload/download all the wheels in the same manner that we do for conda packages (a single tarball for all wheels instead of one per wheel), so we can get rid of support for RAPIDS_PY_WHEEL_NAME. In the PRs for Support dynamic linking between RAPIDS wheels #33 we're currently abusing RAPIDS_PY_WHEEL_NAME to handle the CUDA version, so we need to start adding it for wheels before we can get rid of that variable.
  • tracked in consolidate wheel-building CI jobs #250
• rapids-download-conda-from-s3 automates choosing the output directory, while rapids-download-wheels-from-s3 requires that the caller specify it. We should update the wheel tool to automate that too.
• Update conda jobs to include conda in the name. Currently wheels jobs are e.g. build_wheel_*, whereas conda is just build_cpp.sh etc. That is an artifact of a time when conda was our only produced artifact.
  • tracked in Standardize CI scripts and names #249
• The rapids-wheels-anaconda tool will need to be modified to support upload of cpp wheels.
• We should standardize the names of the CI jobs across repos, conda/wheel build types, and pr.yaml/build.yaml/test.yaml. See also add wheel output kvikio#369
  • tracked in Standardize CI scripts and names #249

I will update this list as more ideas come to mind.

[vyasr]

A separate but related topic that may make sense to address somewhat in parallel is finding a way to make some subsets of the scripts reusable. Some work was previously done to make the CI test scripts usable in other environments (e.g. rapidsai/raft#2165), so when aligning conda/wheels CI and finding ways to maximize shared code (e.g. both calling the same test_* scripts) it might also make sense to see if we can make sure build and test scripts could be used in other environments, such as devcontainers. However, in an ideal world build scripts would be trivial (i.e. just cmake -S . -B build && cmake --build build, or python -m pip install ..., so there might not be much to do for build scripts (and test scripts have already been handled as mentioned above).

[jameslamb]

The rapids-wheels-anaconda tool will need to be modified to support upload of cpp wheels.

Put up a proposal for this in rapidsai/gha-tools#105

[sisodia1701]

This is dependent on C++ wheels work and team will start work on this one, once the C++ work is done.
Moving to backlog.

[jameslamb]

Updates here in early 2026.

We should parallelize test jobs across different Python packages. Wheels generally already do this, while all conda tests typically occur within a single job

There is a mix of this across repos but I think the current state where it's done on a case-by-case basis is fine. I think we should consider this one "no changes needed".

In several repos the conda tests actually run different subsets of test cases for conda vs. wheels, and the division isn't always about the separation between different package names.

For example, this seems to be working well for cuML (just adding more detail on what Vyas already shared above)

  conda-python-tests-singlegpu:
    uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@main
    with:
      script: "ci/test_python_singlegpu.sh"
  conda-python-tests-cudf-pandas-integration:
    uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@main
    with:
      script: "ci/test_python_integration.sh"
  conda-python-tests-dask:
    with:
      script: "ci/test_python_dask.sh"
  conda-python-scikit-learn-accel-tests:
    with:
      script: "ci/test_python_scikit_learn_tests.sh"

(rapidsai/cuml - .github/workflows/pr.yaml)

...
_automatically append the CUDA version to the artifacts produced by the gha-tools for uploading packages _
...
also upload/download all the wheels in the same manner that we do for conda packages (a single tarball for all wheels instead of one per wheel)

This is complex enough that it should be its own sub-issue, but yes it's not done and yes it still should be.

Right now each different wheel is built in its own dedicated CI job, e.g. build-wheel-libraft, build-wheel-pylibraft, build-wheel-raft-dask.

There will be fewer total jobs once #42 is complete, but still more total jobs for the same set of packages than conda, which tend to just have conda-cpp-build (all C++ packages) and conda-python-build (all Python packages).

rapids-download-conda-from-s3 automates choosing the output directory, while rapids-download-wheels-from-s3 requires that the caller specify it. We should update the wheel tool to automate that too.

This is done.

Artifact-handling all moved from S3 to GitHub Actions (https://github.com/rapidsai/build-infra/issues/237), and these gha-tools were modified to adapt to that.

We now have rapids-download-conda-from-github and rapids-download-wheels-from-github which both call into a rapids-download-from-github which defaults to creating a temporary directory with mktemp (overridable with RAPIDS_UNZIP_DIR environment variable.

See:

• https://github.com/rapidsai/gha-tools/blob/211953f03f60af5569a8677a87eb802f7a3c70f5/tools/rapids-download-from-github#L19
• Adding scripts to download and upload artifacts to the Github Artifact Store gha-tools#152

Update conda jobs to include conda in the name. Currently wheels jobs are e.g. build_wheel*, whereas conda is just build_cpp.sh etc. That is an artifact of a time when conda was our only produced artifact._

I think this was meant to be "update conda SCRIPTS" not "jobs"? Either way this is not done.

The rapids-wheels-anaconda tool will need to be modified to support upload of cpp wheels.

This is done. The relevant tools now take an argument with values like cpp or python that are used to create the expected artifact names.

_We should standardize the names of the CI jobs across repos, conda/wheel build types, and pr.yaml/build.yaml/test.yaml. _

Similarly not done. For example RMM has build-wheel-cpp for building librmm while cuDF has build-wheel-libcudf.

Those types of inconsistencies are just historical artifacts and not for any functional purpose, they should be resolved.