[BUG] Title: 2-node Mac cluster with Qwen3.5-397B-A17B-4bit becomes unstable after a few prompts: peer queues full → broken pipe → worker teardown

[tniccum21]

Environment

• EXO commit on both Macs: 7ed46395
• Both nodes on same build
• Model: mlx-community/Qwen3.5-397B-A17B-4bit
• 2-node Mac cluster
• Not intentionally using RDMA
• EXO_FAST_SYNCH=off tested
• Thunderbolt Bridge removed/disabled on MSM2 for latest tests

Hardware

• Node 1: MacBook Pro M5 MAX (128GB)
• Node 2: Mac Studio M2 Ultra (192GB)

What works

• Both nodes start cleanly by themselves
• Model downloads complete successfully on both nodes
• /instance/previews returns valid placements
• Best/only semi-stable manual launch settings so far:
  • Pipeline
  • TCP/IP
  • MLX Ring

With those settings, the model launches and can answer a few prompts.

What fails

After a few prompts, the cluster becomes unstable and the instance fails.

Symptoms:

• Answer gets cut off mid-generation
• Dashboards eventually show FAILED
• MSM2 starts flapping / re-electing
• API on MSM2 goes down

Key observation

It looks like a runtime / control-plane collapse on MSM2 during sustained distributed inference.

Logs from the failing node (MSM2)

[ 2026-03-14 13:40:30.888 | WARNING  | exo.routing.router:_networking_publish:230 ] All peer queues full, dropping message on local_events
...
[ 2026-03-14 13:40:31.234 | WARNING  | logging:handle:1680 ] Failure while closing connection: i/o error: Broken pipe (os error 32)
[ 2026-03-14 13:40:31.439 | INFO     | exo.shared.election:_campaign:197 ] Waiting for other campaign to finish
[ 2026-03-14 13:40:32.881 | INFO     | exo.shared.election:_campaign:194 ] Cancelling other campaign
[ 2026-03-14 13:40:33.164 | INFO     | exo.routing.event_router:_nack_request:150 ] Nack attempt 1: Requesting Event Log from 6539
[ 2026-03-14 13:40:35.120 | INFO     | exo.routing.event_router:_nack_request:150 ] Nack attempt 2: Requesting Event Log from 6539
[ 2026-03-14 13:40:35.883 | INFO     | exo.main:_elect_loop:199 ] Node elected Master - promoting self
[ 2026-03-14 13:40:35.886 | INFO     | exo.master.api:reset:257 ] Resetting API State
[ 2026-03-14 13:40:35.900 | INFO     | exo.master.main:run:97 ] Starting Master
[ 2026-03-14 13:40:35.902 | INFO     | exo.worker.main:run:83 ] Starting Worker
[ 2026-03-14 13:40:35.946 | INFO     | exo.worker.main:run:97 ] Stopping Worker
[ 2026-03-14 13:40:35.946 | INFO     | exo.worker.runner.runner_supervisor:shutdown:118 ] Runner supervisor shutting down
[ 2026-03-14 13:40:35.947 | INFO     | exo.worker.runner.bootstrap:entrypoint:72 ] bye from the runner
[ 2026-03-14 13:40:37.168 | INFO     | exo.worker.runner.runner_supervisor:shutdown:134 ] Runner process succesfully terminated
Comparison with the other node (MBPM5)

At the same timestamp window, MBPM5 does not show the queue flood first. It only shows the aftermath / election transition:

[ 2026-03-14 13:40:32.884 | INFO     | exo.shared.election:_campaign:197 ] Waiting for other campaign to finish
[ 2026-03-14 13:40:35.887 | INFO     | exo.main:_elect_loop:194 ] Node elected Master
[ 2026-03-14 13:40:35.888 | INFO     | exo.master.api:unpause:270 ] Unpausing API

This suggests MSM2 is the node that fails first.

Additional details

/instance/previews showed valid placements for:

Pipeline / MlxRing

Pipeline / MlxJaccl

Tensor / MlxRing

Tensor / MlxJaccl

The JACCL previews referenced rdma_en* interfaces, and earlier previews also showed a 169.254.x.x address on one node. Auto mode seemed to choose bad paths. Manually forcing Pipeline / TCP/IP / MLX Ring got the furthest.

Repro summary

Start EXO on both Macs (same commit)

Use model mlx-community/Qwen3.5-397B-A17B-4bit

Launch manually with:

Pipeline

TCP/IP

MLX Ring

Send a few prompts

After a few requests, MSM2 starts logging:

All peer queues full, dropping message on local_events

Broken pipe

MSM2 self-elects, resets API state, and tears down worker/runner

Cluster ends up failed

Expected

The 2-node instance should remain stable across multiple prompts.

Actual

The cluster works briefly, then control-plane / peer communication appears to collapse on MSM2 under sustained load.

[Evanev7]

what do memory usages look like at the time of failure on the m5 max? the peer queues full is typically a downstream issue of the non-master node dying, and then starting the process of rejoining the cluster.

[tniccum21]

Seems like it was the M2 that failed... I did have RDMA enabled on M2 - which i don't think works as it only has TB4... so I disabled that, but same result. Have tried with other large-ish models with same result.

I have 128G on M5 Max and 192G on M2 Ultra...

Can you recommend a setup that I should use to test? RDMA enabled/disabled? (Assume disabled); networking over Thunderbolt bridge OK (40GB but not RDMA)?

Let me know I'll test any arrangement.

Let me know how you want memory usage reported (what tool is preferred).

[tniccum21]

Starting over fresh: Qwen-3.5-397B-A17B-4bit...
Thunderbolt bridge - disabled

load complete,
M5 shows 92.7GG/128G
M2 shows 138.6G/192G

Advanced options:
Pipeline, TCP/IP

Loads fine, first chat: "Hello"

Python quit unexpectedly.
Memory didn't seem to change
GPU is pegged at 100% on M5, 27% on M2.

Just sitting like that, not responding - probably because python "unexpectedly quite" - i did hit restart, but I'm sure things are broken

Restart both exos
Memory not released on M5. M2 memory released.

[Evanev7]

yeah tb bridge is ok, don't use rdma (it shouldn't work to begin with). memory usage is shown in the dashboard but any tool is fine. gpu pegged at 100% is a failure we've attempted to fix upstream - are you using our version of mlx? it sounds like you might not be.

i'd suggest running exo with nix which will pull in the correct mlx version that fixes this issue.

[tniccum21]

I was able to get nix run .#exo working on both an M5 Max MacBook Pro and an M2 Ultra Mac Studio, but I hit two separate issues along the way.

First, on both M5 Max and M2 Ultra, the Nix path failed immediately with the same flake/MLX error:

error: 'builtins.head' called on an empty list

This came from the flake.nix logic that assumes the mlx entry in uv.lock has a particular shape. On both machines, I was able to get past that by patching flake.nix as follows:

perl -0pi -e 's/p.name == "mlx" && p.source ? git/p.name == "mlx"/g' flake.nix
perl -0pi -e 's/uvLockMlxRev = builtins.elemAt (builtins.split "#" mlxPackage.source.git) 2;/uvLockMlxRev = if mlxPackage ? source && mlxPackage.source ? git then builtins.elemAt (builtins.split "#" mlxPackage.source.git) 2 else "main";/g' flake.nix

After that, nix run .#exo was able to proceed on both systems.

Second, on the M5 Max, the bundled/use-integration macmon path was not compatible. This is already documented upstream in the macmon repo; there is an open PR that fixes M5 support, but it has not been merged yet. I verified that the patched macmon branch itself works on M5 Max, but exo was still erroring in its macmon integration path.

For my use case, I only needed correct memory reporting in the dashboard, so I disabled the macmon monitor and re-enabled the fallback memory polling in info_gatherer.py. With that change, the dashboard reported available memory correctly on the M5 Max.

So the order of fixes was:

Patch flake.nix to get past the MLX / uv.lock Nix failure.

Run nix run .#exo.

Only if on Mac M5 Max (cause of failure per macmon issue list), patch info_gatherer.py to disable macmon and use the fallback memory polling.

After doing that, I was able to load:

Qwen-3.5-397B-A17B-4bit

and it seems to be running well.

If helpful, I can also share the exact info_gatherer.py change I used for the M5 Max workaround.

Getting ~30 t/s with this setup and this giant model.

[Evanev7]

wow - this is a great investigation! i've added a nix patch for macmon - i believe your uv lock has updated so i won't apply the flake changes immediately, but hopefully we can get back on an upstream version of mlx in the future & simplify our flake.

[tniccum21]

Happy to help. I have the following in my lab:
2xDGX Spark
1 M2 Ultra studio (192g)
1 M5 Max (128g)

I can test any/all combinations if you need anything done.

[Evanev7]

we have some attempts to get the spark running in #1705 - its changing quickly at the moment but itd be good to know if you can get the spark and mac to talk to each other over usb-c. we've had a lot of difficulty there, we have some scripts to patch drivers in place but im slightly hoping a newer spark will work slightly more out of the box