[TRACKING] Framework Desktop Support

[Evanev7]

EXO Does not currently support linux unified memory machines like the Framework Desktop out of the box. We should add this ASAP

[AndreiCravtov]

True

[Gulianrdgd]

Hey, I created this pull request for Ollama to allow distributed inference on the framework desktop. ollama/ollama#13087.

However I would like to run EXO, since then we could ideally connect more devices than just our framework desktops.

In my pull request I use the ROCm driver instead of CUDA or Vulkan for the framework desktop. I see that in your planned Tier 2 platform support there is not mention of ROCm? Why is that exactly? (Here is some info on ROCm for AMD Strix Halo https://rocm.docs.amd.com/projects/radeon-ryzen/en/latest/docs/install/installryz/native_linux/install-ryzen.html)

Maybe I can help with some of the challenges?

[AndreiCravtov]

I think given that AMD's equivalent to CUDA is infact ROCm and the unified memory chip on framework desktop is from ADM, it should be ROCm on that list.

But let's see what others say.

[Evanev7]

We generally haven't investigated ROCm vs Vulkan support enough to commit to what kind of support we're going to provide - Vulkan seemed the more general option so it's what we added yesterday - I'll give those docs a read and we can re-evaluate.

[geerlingguy]

Vulkan would work across Intel, AMD, and Nvidia (if for some reason you don't want to use CUDA), and currently has great support across even arm64 and riscv64 architectures too, which ROCm supports... almost not at all :(

[nbritton]

Chasing ROCm is a boondoggle because AMD drops support for their hardware before anyone even has time to develop a software stack around their ecosystem. Compounding on that, Vulkan is already faster than ROCm anyways. So focus your efforts on Vulkan and RDMA ibverbs with ConnectX-4/5/6/7/8 cards (mlx5) with PeerDirect / GPUDirect.

As Jeff already stated Vulkan is hardware agnostic by design, and at some point in the future once Apple EOL their products the Linux community will develop a Vulkan driver for Apple Silicon. Additionally MoltenVK Vulkan Portability implementation already exists as a target on macOS, so you could support 90% of the GPU market (NVIDIA, AMD, Apple, Intel) merely by supporting Vulkan.

[Atomic-Germ]

Wait how have they even had time to drop ROCm already!? And what about the NPU situation?

EDIT: At least on my Ryzen 5 AI 340; it's a lot faster just to ignore everything but the CPU. So, support for the hardware is so bad it's pointless. It does generate a whole lot more heat though.

[triko88]

Vulkan would work across Intel, AMD, and Nvidia (if for some reason you don't want to use CUDA), and currently has great support across even arm64 and riscv64 architectures too, which ROCm supports... almost not at all :(

Vulkan support is a great option. However, as of now, the project only use MLX safetensors, which neither llama.cpp nor Ollama support. Since the major Linux LLM frameworks do not support MLX safetensors, this requires a tailored option.

I recently built a custom inference engine using tinygrad for this project. The earlier versions of exo used tinygrad, so I decided to pick it up. However, this time, I built a deserialiser to normalize and load MLX weights on Omarchy Linux. Then inferred the loaded weights purely with tensor ops, avoiding any device specific hacking. To optimize my build, I fused kernels, attention and MLP layers. Along with enabling TinyJIT, BEAM search and simplified KV cache.

And the result is in this footage. Running Llama 3.2 1B param model (4-bit quant) at upto 65 tok/sec, even with video recording. That too, on AMD RX 6650M, a mobile GPU with limited tinygrad support. To change the inference backend, I simply had to change the environment variable DEV=<backend> to DEV=CL for OpenCL, and DEV=CLANG for CPU. I used DEV=HIP since my GPU doesn't support MMU operations from tinygrad AMD backend. For AMD RDNA 3+ GPUs, this would work better than on my laptop. I tested this with OpenCL too with just an environment variable change, so it should work with CUDA but I can't guarantee it. I haven't tested it.

Link to my branch: https://github.com/triko88/exo/tree/feature/linux-support

screenrecording-2026-03-03_22-19-30.mp4

[Evanev7]

very nice! with the current runner cleanup for continuous batching we're getting close to supporting more backends, it's the cross-backend stuff that's biting atm