Question about Mooncake PD with NVLink on the same node

[ChuanhongLi]

Hi sglang team! We want to use Mooncake nvlink transport for PD disaggregation deployment on the same node.
According to the PR #11321, we should set the following env:

export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True
export MC_FORCE_MNNVL=True

But in the PR #7264, we see that the SGLANG_MOONCAKE_CUSTOM_MEM_POOL is used for cross-node NVLink transport.

Motivation
This PR adds an env var option: SGLANG_MOONCAKE_CUSTOM_MEM_POOL, if it is set to true, mooncake will initialize the kvcache in a custom memory pool to enable cross-node NVLink transport.

This feature requires mooncake-transfer-engine >= 0.3.3.post2

That is to say, if we use PD disaggregation on the same node with mooncake nvlink transport, we just need to set:

export MC_FORCE_MNNVL=True

Right?

Thanks for your help!

[ShangmingCai]

MNNVL complie option is not enabled by default when releasing x86_64 whl to the PyPi (enabled by default for aarch64), so you have to compile it from source, then you can use export MC_FORCE_MNNVL=True to enable cross-node NVLink transport. For the same node, please wait for Mooncake Transfer Engine V1 (a preview version https://github.com/kvcache-ai/Mooncake/tree/tent), current V0 doesn't support it yet.

[ChowXu]

Hi @ShangmingCai , did the Mooncake Transfer Engine support NVLink transport on the same node now ?

[ShangmingCai]

@ChowXu you can try this PR: kvcache-ai/Mooncake#1341

[ChowXu]

thank you @ShangmingCai for the suggest.

however i encountered such errors, do you have any idea about it ?

[2026-01-27 07:09:39] Some requests timed out when bootstrapping, which means prefill instances fail to receive the KV indices from the decode instance of this request. If a greater mean TTFT is acceptable, you can 'export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600' (10 minutes) to relax the timeout condition.
[2026-01-27 07:09:39] Prefill bootstrap failed for request rank=0 req.rid='8ad48693e43949059ac2790a0686317b' req.bootstrap_room=6764019490578504294 with exception KVTransferError(bootstrap_room=6764019490578504294): Request 6764019490578504294 timed out after 300.0s in KVPoll.Bootstrapping
[2026-01-27 07:09:39] INFO:     127.0.0.1:38534 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error

i tried to build the wheel from source code

  cmake .. \
    -DUSE_MNNVL=ON \
    -DUSE_CUDA=ON \
    -DUSE_TCP=ON \
    -DUSE_HTTP=ON \
    -DUSE_INTRA_NVLINK=ON
  make -j$(nproc)

export MC_INTRANODE_NVLINK=1

pd disagg prefill node lunch

python -m sglang.launch_server \
  --model-path Qwen3-32B \
  --page-size 64 \
  --disaggregation-mode prefill \
  --base-gpu-id 6 \
  --port 30000 \
  --tp-size 1 

env

python3 -m sglang.check_env
Python: 3.12.3 (main, Nov  6 2025, 13:44:16) [GCC 13.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-80GB
GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.9, V12.9.86
CUDA Driver Version: 550.144.03
PyTorch: 2.9.1+cu129
sglang: 0.5.7
sgl_kernel: 0.3.20
flashinfer_python: 0.5.3
flashinfer_cubin: 0.5.3
flashinfer_jit_cache: 0.5.3+cu129
triton: 3.5.1
transformers: 4.57.1
torchao: 0.9.0
numpy: 2.4.0
aiohttp: 3.13.2
fastapi: 0.128.0
hf_transfer: 0.1.9
huggingface_hub: 0.36.0
interegular: 0.3.3
modelscope: 1.33.0
orjson: 3.11.5
outlines: 0.1.11
packaging: 25.0
psutil: 7.2.1
pydantic: 2.12.5
python-multipart: 0.0.21
pyzmq: 27.1.0
uvicorn: 0.40.0
uvloop: 0.22.1
vllm: Module Not Found
xgrammar: 0.1.27
openai: 2.6.1
tiktoken: 0.12.0
anthropic: 0.75.0
litellm: Module Not Found
decord2: 3.0.0
NVIDIA Topology:
	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV12	NV12	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU1	NV12	 X 	NV12	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU2	NV12	NV12	 X 	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU3	NV12	NV12	NV12	 X 	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU4	NV12	NV12	NV12	NV12	 X 	NV12	NV12	NV12	24-47,72-95	1		N/A
GPU5	NV12	NV12	NV12	NV12	NV12	 X 	NV12	NV12	24-47,72-95	1		N/A
GPU6	NV12	NV12	NV12	NV12	NV12	NV12	 X 	NV12	24-47,72-95	1		N/A
GPU7	NV12	NV12	NV12	NV12	NV12	NV12	NV12	 X 	24-47,72-95	1		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Hypervisor vendor:: KVM
ulimit soft: 1048576

[ShangmingCai]

CC: @TTThanos

[TTThanos]

thank you @ShangmingCai for the suggest.

however i encountered such errors, do you have any idea about it ?

[2026-01-27 07:09:39] Some requests timed out when bootstrapping, which means prefill instances fail to receive the KV indices from the decode instance of this request. If a greater mean TTFT is acceptable, you can 'export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600' (10 minutes) to relax the timeout condition.
[2026-01-27 07:09:39] Prefill bootstrap failed for request rank=0 req.rid='8ad48693e43949059ac2790a0686317b' req.bootstrap_room=6764019490578504294 with exception KVTransferError(bootstrap_room=6764019490578504294): Request 6764019490578504294 timed out after 300.0s in KVPoll.Bootstrapping
[2026-01-27 07:09:39] INFO:     127.0.0.1:38534 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error

i tried to build the wheel from source code

  cmake .. \
    -DUSE_MNNVL=ON \
    -DUSE_CUDA=ON \
    -DUSE_TCP=ON \
    -DUSE_HTTP=ON \
    -DUSE_INTRA_NVLINK=ON
  make -j$(nproc)

export MC_INTRANODE_NVLINK=1

pd disagg prefill node lunch

python -m sglang.launch_server \
  --model-path Qwen3-32B \
  --page-size 64 \
  --disaggregation-mode prefill \
  --base-gpu-id 6 \
  --port 30000 \
  --tp-size 1 

env

python3 -m sglang.check_env
Python: 3.12.3 (main, Nov  6 2025, 13:44:16) [GCC 13.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-80GB
GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.9, V12.9.86
CUDA Driver Version: 550.144.03
PyTorch: 2.9.1+cu129
sglang: 0.5.7
sgl_kernel: 0.3.20
flashinfer_python: 0.5.3
flashinfer_cubin: 0.5.3
flashinfer_jit_cache: 0.5.3+cu129
triton: 3.5.1
transformers: 4.57.1
torchao: 0.9.0
numpy: 2.4.0
aiohttp: 3.13.2
fastapi: 0.128.0
hf_transfer: 0.1.9
huggingface_hub: 0.36.0
interegular: 0.3.3
modelscope: 1.33.0
orjson: 3.11.5
outlines: 0.1.11
packaging: 25.0
psutil: 7.2.1
pydantic: 2.12.5
python-multipart: 0.0.21
pyzmq: 27.1.0
uvicorn: 0.40.0
uvloop: 0.22.1
vllm: Module Not Found
xgrammar: 0.1.27
openai: 2.6.1
tiktoken: 0.12.0
anthropic: 0.75.0
litellm: Module Not Found
decord2: 3.0.0
NVIDIA Topology:
	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV12	NV12	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU1	NV12	 X 	NV12	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU2	NV12	NV12	 X 	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU3	NV12	NV12	NV12	 X 	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU4	NV12	NV12	NV12	NV12	 X 	NV12	NV12	NV12	24-47,72-95	1		N/A
GPU5	NV12	NV12	NV12	NV12	NV12	 X 	NV12	NV12	24-47,72-95	1		N/A
GPU6	NV12	NV12	NV12	NV12	NV12	NV12	 X 	NV12	24-47,72-95	1		N/A
GPU7	NV12	NV12	NV12	NV12	NV12	NV12	NV12	 X 	24-47,72-95	1		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Hypervisor vendor:: KVM
ulimit soft: 1048576

Some problemes has been found, working on them

[TTThanos]

thank you @ShangmingCai for the suggest.

however i encountered such errors, do you have any idea about it ?

[2026-01-27 07:09:39] Some requests timed out when bootstrapping, which means prefill instances fail to receive the KV indices from the decode instance of this request. If a greater mean TTFT is acceptable, you can 'export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600' (10 minutes) to relax the timeout condition.
[2026-01-27 07:09:39] Prefill bootstrap failed for request rank=0 req.rid='8ad48693e43949059ac2790a0686317b' req.bootstrap_room=6764019490578504294 with exception KVTransferError(bootstrap_room=6764019490578504294): Request 6764019490578504294 timed out after 300.0s in KVPoll.Bootstrapping
[2026-01-27 07:09:39] INFO:     127.0.0.1:38534 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error

i tried to build the wheel from source code

  cmake .. \
    -DUSE_MNNVL=ON \
    -DUSE_CUDA=ON \
    -DUSE_TCP=ON \
    -DUSE_HTTP=ON \
    -DUSE_INTRA_NVLINK=ON
  make -j$(nproc)

export MC_INTRANODE_NVLINK=1

pd disagg prefill node lunch

python -m sglang.launch_server \
  --model-path Qwen3-32B \
  --page-size 64 \
  --disaggregation-mode prefill \
  --base-gpu-id 6 \
  --port 30000 \
  --tp-size 1 

env

python3 -m sglang.check_env
Python: 3.12.3 (main, Nov  6 2025, 13:44:16) [GCC 13.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA A100-SXM4-80GB
GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.9, V12.9.86
CUDA Driver Version: 550.144.03
PyTorch: 2.9.1+cu129
sglang: 0.5.7
sgl_kernel: 0.3.20
flashinfer_python: 0.5.3
flashinfer_cubin: 0.5.3
flashinfer_jit_cache: 0.5.3+cu129
triton: 3.5.1
transformers: 4.57.1
torchao: 0.9.0
numpy: 2.4.0
aiohttp: 3.13.2
fastapi: 0.128.0
hf_transfer: 0.1.9
huggingface_hub: 0.36.0
interegular: 0.3.3
modelscope: 1.33.0
orjson: 3.11.5
outlines: 0.1.11
packaging: 25.0
psutil: 7.2.1
pydantic: 2.12.5
python-multipart: 0.0.21
pyzmq: 27.1.0
uvicorn: 0.40.0
uvloop: 0.22.1
vllm: Module Not Found
xgrammar: 0.1.27
openai: 2.6.1
tiktoken: 0.12.0
anthropic: 0.75.0
litellm: Module Not Found
decord2: 3.0.0
NVIDIA Topology:
	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV12	NV12	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU1	NV12	 X 	NV12	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU2	NV12	NV12	 X 	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU3	NV12	NV12	NV12	 X 	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU4	NV12	NV12	NV12	NV12	 X 	NV12	NV12	NV12	24-47,72-95	1		N/A
GPU5	NV12	NV12	NV12	NV12	NV12	 X 	NV12	NV12	24-47,72-95	1		N/A
GPU6	NV12	NV12	NV12	NV12	NV12	NV12	 X 	NV12	24-47,72-95	1		N/A
GPU7	NV12	NV12	NV12	NV12	NV12	NV12	NV12	 X 	24-47,72-95	1		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Hypervisor vendor:: KVM
ulimit soft: 1048576

This problem has been solved in Recent version Mooncake. You can try to compile Mooncake using
cmake .. -DUSE_CUDA=ON -DUSE_ETCD=ON -DUSE_TENT=ON -DUSE_INTRA_NVLINK=ON -DUSE_MNNVL=ON
and use following command to start your SGlang server:
For prefill Node
SGLANG_MOONCAKE_CUSTOM_MEM_POOL=true MC_LOG_LEVEL=INFO SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK MC_TE_METRIC=true MC_INTRANODE_NVLINK=true SGLANG_TORCH_PROFILER_DIR=/root/Yaozhong_hiecache/profile/ python3 -m sglang.launch_server \ --model-path ${model_path} \ --tp 4 \ --mem-fraction-static 0.85 \ --disaggregation-mode prefill \ --port 7001 \ --watchdog-timeout 1000000 --decode-log-interval 1 >/root/LYZ_hicache/log/${FILE_NAME_PREFIX}.log 2>&1
and for decode node
export CUDA_VISIBLE_DEVICES=4,5,6,7 export MC_TE_METRIC=true export MC_INTRANODE_NVLINK=true SGLANG_MOONCAKE_CUSTOM_MEM_POOL=true SGLANG_MOONCAKE_CUSTOM_MEM_POOL=INTRA_NODE_NVLINK MC_LOG_LEVEL=INFO SGLANG_TORCH_PROFILER_DIR=/root/Yaozhong_hiecache/profile/ python3 -m sglang.launch_server \ --model-path ${model_path} \ --tp 4 \ --mem-fraction-static 0.85 \ --base-gpu-id 0 \ --disaggregation-mode decode \ --port 7002 \ --watchdog-timeout 1000000 --decode-log-interval 1 >/root/LYZ_hicache/log/${FILE_NAME_PREFIX}.log 2>&1