[Roadmap] DeepSeek v3.2 Optimization

[Fridge003]

Background

There has been an increasing need of deployment for DeepSeek V3.2 and DeepSeek V3.2 Speciale since their release. However, there are still some functionality/performance gap between DeepSeek V3.2 and DeepSeek V3.1.

Optimization Items

Parallelism

• Initial support for Pure TP & Partial DP Attention [DeepSeekV3.2] Enable pure TP & Partial DP Attention #13646
• Initial support for CP (CP + DP + EP, not compatible with TP) (1/n)support context parallel with deepseekv3.2-DSA #12065
• CP optimization with fused moe/multi-batch/fp8 kvcache [DeepSeek v3.2] opt Context Parallelism: support fused moe, multi batch and fp8 kvcache #13959
• TBO support fix ds3.2 nsa backend prefill TBO #14901
• PP support & Optimization [NSA] Fix NSA backend assertion error when running DeepSeek-V3.2 PP with radix-cache #15086
• CP+PP+TP [Feature] DeepSeek V3.2 Support CP8+PP2+TP8 #15358 [DeepSeek 3.2] Support and optimize pipeline parallelis when context pipeline enabled #16380

Kernel/Algorithm Optimization (Prior roadmap #11989)

• DeepGeMM fp8_mqa_logits upgrade [sgl-kernel] chore: update deepgemm version #13402
• Update DeepGemm sgl_release branch to include commits from deepseek-ai/DeepGEMM@main...nv_dev [sgl-kernel] chore: update deepgemm version #13402
• Integrate FP8 per tensor sparse MLA kernel from trtllm(flashinfer) Nsa trtllm mla sparse fp8 support with Deepseek v3.2 NVFP4 #18389
• Integrate BF16 per tensor sparse MLA kernel from trtllm(flashinfer) [DeepSeek V3.2] Enable trtllm NSA with bf16 kvcache #16758
• [Decode] Optimize dual stream in Indexer [Deepseek V3.2] Optimize use of dual_stream in nsa_indexer/attention #13546 [DSv32] Overlap indexer weights_proj during dual_stream decode #16637
• [Decode] Move deep_gemm.get_paged_mqa_logits_metadata to init time as metadata [DSv32] Move deep_gemm.get_paged_mqa_logits_metadata to init time as metadata #15040
• [Decode] Optimize _get_topk_paged where there are a lot of small kernels [Perf][DeepSeekV3.2][Decode] Reduce small-kernel overhead in _get_topk_paged #15104
• [Prefill] Optimize _get_topk_ragged where there are a lot of small kernels. optimize get_topk_ragged by fusing get k and k_scale triton kernel #16043
• [Prefill] Optimize with masked MHA kernels for sequence with ISL > 2k [Bug] Performance Regression: DeepSeek V3.2 Prefill Throughput is 1/3 of V3.1 in Disaggregation Mode #14498 feat: try impl masked mha for sm90a sgl-flash-attn#24
• Update flashmla to nv_dev branch: [Feature] Update flashmla to nv_dev branch #15211
• Allreduce+norm fusion (completed) [Deepseek V3.2] Enable TRTLLM Allreduce Fusion #15310
• Support TP for Indexer when DP is not enabled
• Hi-cache support [HiCache]: Support DeepSeek v32 cpu offloading #17415

MTP

• MTP (spec-v1) initial support [Spec Decoding] Support MTP for dsv3.2 #11652
• Support pure TP+MTP [DeepSeekV3.2] Add pure TP+MTP test #15088
• Support Spec-V2 with overlap scheduler [Deepseek V3.2] Support Overlap Spec + NSA #15307
• Reuse metadata for multi-step MTP [Performance] optimize NSA backend metadata computation for multi-step speculative decoding #14781
• Enable nextn = 2/4 in deep_gemm.fp8_paged_mqa_logits, which is faster than the current implementation which uses the kernel with nextn = 1 regardless of mtp size.
• Apply decode dsa kernels in target verify and draft extend [DeepSeek v3.2] Opt MTP decode cuda batch sizes and nsa implementation #16961

Others

• Support of fp4 checkpoint after its release (on Blackwell) [Feature] Optimization of DeepSeek V3.2 NVFP4 checkpoint #17655
• Compatibility with PD disagg (MTP, TP, DP, EP) [Bug] DeepSeek V3.2 PD decode TP+DP+EP +MTP launch failed #14496
• ROCM optimization
• NPU optimization

Related resources

• Model link: DeepSeek V3.2, DeepSeek V3.2 Speciale
• Official document for DeepSeek V3.2 usage on SGLang: https://docs.sglang.io/basic_usage/deepseek_v32.html
• Bug tracking issue [Track] DeepSeek v3.2 bug tracking #14511
• Initial tracking issues for DeepSeek V3.2: [Tracking] DeepSeek-V3.2-Exp Day 0 Support #11060 [Feature] DeepSeek-V3.2-Exp Day 0 Support on NPU #11100
• Prior issue for DSA kernel optimization: [Feature] NSA optimization roadmap #11989

Profiling command example

export SGLANG_TORCH_PROFILER_DIR=/sgl-workspace/sglang/profile/
python3 -m sglang.launch_server   --model-path deepseek-ai/DeepSeek-V3.2   --trust-remote-code   --tp-size 8 --dp-size 8 --enable-dp-attention   --tool-call-parser deepseekv32   --reasoning-parser deepseek-v3
# bs1
python3 -m sglang.bench_serving --model deepseek-ai/DeepSeek-V3.2 --dataset-name random --backend sglang-oai --random-range-ratio 1 --random-input-len 1200 --random-output-len 20 --max-concurrency 1 --num-prompts 5 --profile
# bs32
python3 -m sglang.bench_serving --model deepseek-ai/DeepSeek-V3.2  --dataset-name random --backend sglang-oai --random-range-ratio 1 --random-input-len 1200 --random-output-len 20 --max-concurrency 32 --num-prompts 32 --profile

Long context performance

Here is the performance comparison between DeepSeek V3.1 and V3.2 on long context lengths. (Collected by @XucSh)

From the figure, we can see that DeepSeek V3.2 shows advantage under long context like 32k, and PP helps a lot with latency.

Benchmark data (Updated on 12/11)

Here is the nightly performance data collected on 01/08 (8*B200, link):

deepseek-ai/DeepSeek-V3.2 [8-gpu-b200] (DP8)

batch size	input len	latency (s)	input throughput (tok/s)	output throughput (tok/s)	acc length	ITL (ms)	input cost ($/1M)	output cost ($/1M)	profile (extend)	profile (decode)
1	4096	10.41	7003.29	52.13	n/a	19.18	0.11	10.66	trace	trace
8	4096	11.20	32024.01	402.37	n/a	19.88	0.02	1.38	trace	trace
16	4096	14.49	42795.90	632.33	n/a	25.30	0.02	0.88	trace	trace
64	4096	19.70	43666.88	2392.64	n/a	26.75	0.02	0.23	trace	trace

deepseek-ai/DeepSeek-V3.2 [8-gpu-b200] (DP8+MTP)

batch size	input len	latency (s)	input throughput (tok/s)	output throughput (tok/s)	acc length	ITL (ms)	input cost ($/1M)	output cost ($/1M)	profile (extend)	profile (decode)
1	4096	5.98	10752.06	91.40	n/a	10.94	0.07	6.08	trace	trace
8	4096	7.44	40049.51	618.72	2.67	12.93	0.02	0.90	trace	trace
16	4096	7.49	41491.37	1386.14	3.09	11.54	0.02	0.40	trace	trace

deepseek-ai/DeepSeek-V3.2 [8-gpu-b200] (TP8)

batch size	input len	latency (s)	input throughput (tok/s)	output throughput (tok/s)	acc length	ITL (ms)	input cost ($/1M)	output cost ($/1M)	profile (extend)	profile (decode)
1	4096	7.23	4634.57	80.62	n/a	12.40	0.17	6.89	trace	trace
8	4096	10.20	15694.80	504.66	n/a	15.85	0.05	1.10	trace	trace
16	4096	13.44	15967.91	877.79	n/a	18.23	0.05	0.63	trace	trace
64	4096	32.10	16129.85	2067.71	n/a	30.95	0.05	0.27	trace	trace

deepseek-ai/DeepSeek-V3.2 [8-gpu-b200] (TP8+MTP)

batch size	input len	latency (s)	input throughput (tok/s)	output throughput (tok/s)	acc length	ITL (ms)	input cost ($/1M)	output cost ($/1M)	profile (extend)	profile (decode)
1	4096	4.36	9093.46	130.92	3.07	7.64	0.09	4.24	trace	trace
8	4096	7.83	15420.75	717.38	3.0	11.15	0.05	0.77	trace	trace
16	4096	10.23	15590.51	1359.11	3.27	11.77	0.05	0.41	trace	trace

deepseek-ai/DeepSeek-V3.1 [8-gpu-b200] (TP8)

batch size	input len	latency (s)	input throughput (tok/s)	output throughput (tok/s)	acc length	ITL (ms)	input cost ($/1M)	output cost ($/1M)	profile (extend)	profile (decode)
1	4096	4.16	22590.03	128.60	n/a	7.78	0.04	4.32	trace	trace
8	4096	6.58	32987.73	733.29	n/a	10.91	0.02	0.76	trace	trace
16	4096	8.92	40294.88	1123.31	n/a	14.24	0.02	0.49	trace	trace
64	4096	18.83	41034.27	2632.78	n/a	24.31	0.02	0.21	trace	trace

deepseek-ai/DeepSeek-V3.1 [8-gpu-b200] (TP8+MTP)

batch size	input len	latency (s)	input throughput (tok/s)	output throughput (tok/s)	acc length	ITL (ms)	input cost ($/1M)	output cost ($/1M)	profile (extend)	profile (decode)
1	4096	2.27	26147.32	241.75	3.05	4.14	0.03	2.30	trace	trace
8	4096	5.78	37656.27	834.55	2.64	9.59	0.02	0.67	trace	trace
16	4096	8.00	38651.86	1299.46	2.56	12.31	0.02	0.43	trace	trace

These results show a huge space for performance improvement on DeepSeek V3.2 model families. On some common low latency cases (bs=1, isl=4k), the performance of V3.2 with TP8 (80 tps) is ~37% slower than the case of V3.1(128 tps) with TP8.

[jxz542189]

When deepseek v3.2 will function call streaming be supported? #14711。This PR（https://github.com/sgl-project/sglang/pull/14750） only implements parameter-level streaming but does not achieve full streaming

[thqq479]

deepseek-ai/DeepSeek-V3.2-Exp (tp8)

1 4096 8.79 3503.99 67.20 n/a 14.88 0.23 8.27
1 4096 7.92 13081.30 67.33 n/a 14.85 0.06 8.25

Why are the test results so different here for the same input? @Fridge003

[Fridge003]

@thqq479 Nice catch...Just checked the results from yesterday. Only DeepSeek V3.2 + pure tp 8 will cause such behavior.
Do you have any clues @YAMY1234

[YAMY1234]

@Fridge003 Yeah this is quite strange especially when bs=1, what is the difference between the first line and the second line here? I saw the bs and inputlen are the same. Also sems I cannot open the trace file listed in https://github.com/sgl-project/sglang/actions/runs/20156061438 for example this trace

[Fridge003]

@YAMY1234 I think there is no difference between the first line and the second line.
I also checked that the radix cache is flushed before running the second line