[CI] Widespread cudaErrorInvalidDevice failure and Segmentation Faults

[csadorf]

Summary

Widespread CI failures across multiple PRs, test types, CUDA versions, and Python versions. All failures share the same underlying CUDA error: cudaErrorInvalidDevice: invalid device ordinal.

Failing jobs: conda-cpp-tests, conda-notebook-tests, docs-build, conda-python-tests-singlegpu, etc.

Example failures:

• https://github.com/rapidsai/cuml/actions/runs/20931777189/job/60153661146?pr=7661
• https://github.com/rapidsai/cuml/actions/runs/20931777189/job/60153661356?pr=7661
• https://github.com/rapidsai/cuml/actions/runs/20934571592/job/60154568407?pr=7662
• and many others

Common Factors

• GPU: all GPUs
• Error: cudaErrorInvalidDevice: invalid device ordinal

Error Messages

RuntimeError: CUDA error encountered at: file=***: call='cudaPeekAtLastError()', Reason=cudaErrorInvalidDevice:invalid device ordinal

Root Cause Analysis

[csadorf]

Possible root cause: rapidsai/rmm#2201

[jameslamb]

If that RMM PR ends up being the root cause, we should be close to a resolution. All of RAPIDS now has new 26.02 nightlies using the new RMM ABI.

The last piece needed for cuML should be new xgboost packages, which I've triggered with rapidsai/xgboost-feedstock#121 (build link: https://github.com/rapidsai/xgboost-feedstock/actions/runs/20942886001)

[csadorf]

The remaining segmentation faults are now tracked in #7678 .