Add has_rocm for OpenMPI

[avik-pal]

See https://docs.open-mpi.org/en/main/man-openmpi/man3/MPIX_Query_rocm_support.3.html#mpix-query-rocm-support

Seems to be available from v5 of OpenMPI

[vchuravy]

We probably need to use MPI_LIBRARY_VERSION to check that this is indeed v5

[avik-pal]

The current compat is on 4. Should we hold it off till newer JLLs are supported?

[giordano]

Someone will have to debug what's the problem with OpenMPI 5: #789 (comment)

[vchuravy]

Someone could already run with a local version of OpenMPI 5 before we get around to jll

[giordano]

If that helps for testing, here's a docker container with openmpi v5.0.2: https://github.com/JuliaParallel/github-actions-buildcache/pkgs/container/github-actions-buildcache/183303325?tag=openmpi-5.0.2-wufrzwmzj3h642rou7vsrphzpysokjui.spack

[giordano]

For the record, MPI.jl tests pass in that container, I'll open a PR to add that to the matrix so that we test OpenMPI v5. We still need to figure out what was the problem with the JLL.

[giordano]

@avik-pal with #826 now merged we actually have coverage for OpenMPI v5 on CI (although not with an AMD GPU available...) so you can do a test like

Suggested change

	@static if MPI_LIBRARY == "OpenMPI"
	@static if MPI_LIBRARY == "OpenMPI" && MPI_LIBRARY_VERSION >= v"5"

as suggested above by @vchuravy and we can at least ensure the ccall is working correctly.

[avik-pal]

Tested locally that ccall works by bumping the jll compat to 5

julia> using MPI

julia> MPI.has_cuda()
false

julia> MPI.has_rocm()
false

[giordano]

Can also document has_rocm in https://juliaparallel.org/MPI.jl/stable/knownissues/#Hints-to-ensure-ROCm-aware-MPI-to-be-functional, like has_cuda is documented in the CUDA section.

[giordano]

@avik-pal are you interested in moving this forward?

[avik-pal]

Completely fell off my radar, I will finish this over the weekend

[JBlaschke]

Once this is merged, I can implement has_gpu; has_cuda and has_rocm for Cray MPICH

[luraess]

Maybe one could add a sentence similar to CUDA (https://github.com/JuliaParallel/MPI.jl/blob/master/docs/src/usage.md)

If using OpenMPI, the status of CUDA support can be checked via the [MPI.has_cuda()](https://github.com/JuliaParallel/MPI.jl/blob/master/docs/src/@ref) function.

in the ROCm-aware section of the docs?

[avik-pal]

Maybe one could add a sentence similar to CUDA (master/docs/src/usage.md)

If using OpenMPI, the status of CUDA support can be checked via the [MPI.has_cuda()](https://github.com/JuliaParallel/MPI.jl/blob/master/docs/src/@ref) function.

in the ROCm-aware section of the docs?

done

[giordano]

Thanks!

[giordano]

Can also document has_rocm in https://juliaparallel.org/MPI.jl/stable/knownissues/#Hints-to-ensure-ROCm-aware-MPI-to-be-functional, like has_cuda is documented in the CUDA section.

Last documentation request and the we should be good to go ☝️

[avik-pal]

Updated the docs

[vchuravy]

Can you erase on master?

[avik-pal]

Can you erase on master?

What do you want me to erase?

[giordano]

I presume that was meant to be "rebase".