[BUG] RDMA stopped working all of the sudden

[leanzero-srl]

Describe the bug

Tried rebooting machines, move the cable from one to the other. Everytime I try to launch an instance it immediatley fails during the prepare phase. Also worked fine until like an hour ago...

To Reproduce

Steps to reproduce the behavior:

1. Connect over TB5
   2.Open web dashboard
   3.Launch an instance

Expected behavior

Crashes on preparing phase

Actual behavior

Launches.

Environment

• macOS Version: 26.3.1
• EXO Version: 1.0.68
• Hardware:
  • Device 1: Mac studio M3 ultra 96gb
  • Device 2: macbook m4 max 128gb
  • Additional devices:
• Interconnection:
  • TB 5 cable

Additional context

Add any other context about the problem here.

[leanzero-srl]

exo.log

[leanzero-srl]

I am not even sure what I did to make it work again, I removed, rebooted ,a combo of it all and it started to work.

[AlexCheema]

I am not even sure what I did to make it work again, I removed, rebooted ,a combo of it all and it started to work.

I see in your logs that the issue was an upstream jaccl error in MLX. If you could report this upstream to the MLX repo that would be great.

[ 2026-03-12 13:32:00.441 | WARNING  | exo.worker.runner.bootstrap:entrypoint:44 ] Runner 3abbeef2-3ac6-4f1b-8076-ca2f03c9956b crashed with critical exception [jaccl] Changing queue pair to RTR failed with errno 22
Traceback (most recent call last):

  File "__main__.py", line 38, in <module>

  File "pyi_rth_multiprocessing.py", line 48, in _freeze_support

  File "multiprocessing/spawn.py", line 122, in spawn_main

  File "multiprocessing/spawn.py", line 135, in _main

  File "multiprocessing/process.py", line 313, in _bootstrap

  File "multiprocessing/process.py", line 108, in run

> File "exo/worker/runner/bootstrap.py", line 39, in entrypoint

  File "exo/worker/runner/llm_inference/runner.py", line 145, in main

  File "exo/worker/engines/mlx/utils_mlx.py", line 169, in initialize_mlx

  File "exo/worker/engines/mlx/utils_mlx.py", line 148, in mlx_distributed_init

ValueError: [jaccl] Changing queue pair to RTR failed with errno 22

We're not sure why this happens but seems pretty random. Often disconnecting and reconnecting the Thunderbolt 5 cable fixes it.

We should be displaying troubleshooting guidance to the user in the dashboard, suggesting that they disconnect and reconnect the Thunderbolt 5 cables. I've created an issue for that here: #1712