Deepseek-R1 MTP poor performance

[jokerwyt]

I expect the time we spend on "verify" part should be close to a normal decode forward (less than 100ms, my setting is bs=16 and ctx=12k), but now it takes about 400ms. It slows down my output throughput severely. Seems like a kernel performance issue?

decoding with mtp profile:

normal decode profile:

The commit I test:
commit 4a05bdf (gh/main)
Author: Lianmin Zheng lianminzheng@gmail.com
Date: Sun Mar 9 18:53:33 2025 -0700

Revert "Check eagle server args" (#4242)

Originally posted by @jokerwyt in #3582 (comment)

[shuaills]

cc @merrymercy

[jokerwyt]

@shuaills It turns out that --enable-flashinfer-mla comes to rescue. With that option, verification stage costs 100ms now. But that option doesn't support topk>1 yet which is important for EAGLE speculative algorithm.

[hebiao064]

@jokerwyt Hi have you tried topk = 1 for FA3?
From our observation, it should be on par with flashinfer-mla.

For top k > 1 case, we do noticed that the latency is not well optimized for FA3 and we are working on it.

[hebiao064]

And it would be nice if you can provide the repro steps.

This is what I did:

python3 -m sglang.launch_server --model /shared/public/elr-models/deepseek-ai/DeepSeek-V3--speculative-algo EAGLE --speculative-draft /shared/public/models/deepseek-eagle/DeepSeek-V3-NextN/ --speculative-num-steps 5 --speculative-eagle-topk 4 --speculative-num-draft-tokens 8  --trust-remote-code   --mem-fraction-static 0.7  --cuda-graph-max-bs 16  --tp 8

python3 -m sglang.bench_serving --backend sglang-oai --num-prompts 5 --request-rate 10 --dataset-name random --random-input-len 10000 --random-output-len 50 --random-range-ratio 1 --profile

and draft decode took 2.3ms and verify took 39ms

[jokerwyt]

@hebiao064 My result seems outdated since the result has improved a lot in the past few months. I will present here if we have any progress on that.