[BUG] DGX Spark does not find other nodes

[nsitnov]

Describe the bug

My DGX Spark does not find two other machines nodes

To Reproduce

Steps to reproduce the behavior:

1. Mac nodes are self discovered but did not see DGX Spark node
2. All of them connected on unmanaged 10Gbit Tp-Link switch
3. All running lates version of EXO

Expected behavior

DGX Spark unreachable on local network

Actual behavior

A clear and concise description of what actually happened.

Environment

• macOS Version: 26.3 (25D125)
• EXO Version: ba611f9
• Hardware:
  • Device 1: Mac Studio M3 Ultra 512GB RAM
  • Device 2: Mac mini M4 16GB RAM
  • Additional devices: DGX Spark
• Interconnection:
  • (e.g., Thunderbolt 4 cable between Device 1 and 2)
  • (e.g., WiFi 6 for Device 3)
  • 10GbE Ethernet between all devices

Additional context

DGX Spark version - DGX Spark: commit ba611f9, version 0.3.68. b343c24112fc9334a2ada634f84a2783bdca03551b0b7282f62b348b93fe9504

[nsitnov]

(exo-venv) nsitnov@spark-7af6:/exo$ git describe --tags
v1.0.68
(exo-venv) nsitnov@spark-7af6:/exo$

and still does not discover Mac nodes

[nsitnov]

Detailed debugging findings — mDNS discovery fails between DGX Spark (Linux aarch64) and Mac nodes

Setup:

• Mac Studio M3 Ultra 512GB — 192.168.1.136 — EXO.app v1.0.68
• MacBook Air M4 Pro 48GB — 192.168.1.92 — EXO.app v1.0.68
• Mac mini M4 16GB — 192.168.1.240 — EXO.app v1.0.68
• DGX Spark (Grace Blackwell, aarch64) — 192.168.1.200 — EXO source v1.0.68 (commit ba611f9)
• All on same subnet 192.168.1.0/24, 10GbE unmanaged TP-Link switch

Network connectivity — CONFIRMED WORKING:

• ping works both directions between Spark and all Macs
• nc -zv 192.168.1.136 49578 (Mac Studio libp2p port) — succeeded from Spark
• nc -zv 192.168.1.200 46619 (Spark libp2p port) — succeeded from Mac Studio
• curl http://192.168.1.200:52415 from Mac — works (returns EXO dashboard HTML)
• Firewall disabled on both sides (ufw inactive on Spark, macOS firewall disabled)

mDNS discovery — THE PROBLEM:

• DGX Spark does send _p2p._udp.local. mDNS packets (confirmed via tcpdump -i enP7s7 -n src 192.168.1.200 and udp port 5353)
• Mac nodes do NOT send _p2p._udp.local. mDNS packets (confirmed via tcpdump -i enP7s7 udp port 5353 — only Apple Bonjour services visible: _companion-link, _airplay, _raop)
• avahi-browse on Spark sees all Mac devices via standard Bonjour but no EXO/libp2p services

libp2p pnet pre-shared key:

• Both sides use NETWORK_VERSION = "v0.0.1" with EXO_LIBP2P_NAMESPACE: None
• Spark binary contains exo_discovery_network and v0.0.1 strings (confirmed via strings)
• Cluster Namespace on Mac EXO.app is set to "default" (empty)

Docker interference (partial):

• Docker bridge networks (docker0, br-*) on Spark caused EXO to bind UDP sockets to 172.17.0.1 and 172.18.0.1 instead of 192.168.1.200
• Stopping Docker (systemctl stop docker docker.socket) fixed the bind issue — EXO then correctly binds to 192.168.1.200
• However, mDNS discovery still fails because Mac nodes never send _p2p mDNS responses

Root cause hypothesis:
The three Mac nodes discover each other but NOT the Linux node. Mac EXO.app likely uses a macOS-specific discovery mechanism (possibly native Bonjour/DNS-SD or RDMA-based Thunderbolt discovery) rather than standard libp2p mDNS. The /state endpoint shows all Mac nodes have nodeThunderbolt and nodeRdmaCtl entries — suggesting Mac-to-Mac discovery may happen over a different channel.

Environment details (Spark):

Kernel: 6.17.0-1008-nvidia (aarch64)
OS: DGX OS (Ubuntu 24.04 based)
Python: 3.13.12
Rust: 1.96.0-nightly
exo_pyo3_bindings: installed and working
avahi-daemon: active and running

---

[Evanev7]

we use standard libp2p mDNS which uses macos bonjour. iirc the default namespace on the app is the version number - try using the same custom namespace for both.

[nsitnov]

Solution Found ✅

After extensive debugging, I identified two root causes for DGX Spark not finding other nodes:

Problem 1: Rust networking module not compiled on DGX Spark

The exo_pyo3_bindings Rust module (which contains all libp2p networking, mDNS discovery, and pnet private network logic) was not compiled on DGX Spark. On macOS it comes pre-built, but on aarch64 Linux (DGX OS) it requires a manual build. Without it, EXO runs in a degraded mode — it sends mDNS packets but cannot establish peer connections.

Fix:

# Install Rust if not present:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source ~/.cargo/env

# Build the Rust networking module:
cd ~/exo/rust
cargo build --release

# Install pyo3 bindings into EXO's venv:
cd ~/exo
uv pip install -e rust/exo_pyo3_bindings/

Problem 2: pnet pre-shared key mismatch

EXO uses a libp2p private network (pnet) with a pre-shared key derived from SHA3-256("exo_discovery_network" + NETWORK_VERSION). Even with the Rust module compiled, the DGX Spark and Mac nodes may generate different keys if their builds differ slightly. The fix is to set the same custom namespace on all nodes via the EXO_LIBP2P_NAMESPACE environment variable (which overrides NETWORK_VERSION in the key derivation — see rust/networking/src/swarm.rs):

# On EVERY node (DGX Spark, Mac Studio, Mac mini, etc.):
EXO_LIBP2P_NAMESPACE=my-cluster uv run exo

Result

After applying both fixes, my DGX Spark (128GB) successfully joined the cluster with Mac Studio M3 Ultra (512GB) and other Mac nodes. The DGX Spark appears in the EXO dashboard and participates in distributed inference.

Suggestion for EXO team

1. The Rust module build should be automated for Linux/aarch64 during uv sync or first run — right now it silently fails and EXO appears to work but can't connect to peers.
2. A clearer error message when the Rust networking module is missing would save hours of debugging.
3. The EXO_LIBP2P_NAMESPACE env var should be documented, as it's essential for heterogeneous clusters.

Environment

• DGX Spark (DGX OS, Ubuntu-based, aarch64)
• Mac Studio M3 Ultra 512GB (macOS 26.3.1)
• EXO v1.0.68 (commit ba611f9)
• All connected via 10GbE unmanaged TP-Link switch