[Bug] Deepseek 3.2 Low Quality sglang:v0.5.6

[pdasgup]

Checklist

• I searched related issues but found no solution.
• The bug persists in the latest version.
• Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
• If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
• Please use English. Otherwise, it will be closed.

Describe the bug

Seeing low quality on mmlu_pro_engineering and mmlu_pro with Deepseek 3.2 on 8xB200. Quality is much lower for this eval set than Deepseek 3.1 (~0.6883).

Reproduction

Image: lmsysorg/sglang:v0.5.6

python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.2 \
  --trust-remote-code \
  --tp-size 8 --dp-size 8 --enable-dp-attention \
  --tool-call-parser deepseekv32 \
  --reasoning-parser deepseek-v3 --port=8000 --host=0.0.0.0 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

eval run with https://github.com/EleutherAI/lm-evaluation-harness

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
git checkout v0.4.9
pip install -e .

lm_eval --apply_chat_template --model local-chat-completions --tasks mmlu_pro_engineering --model_args model=deepseek-ai/DeepSeek-V3.2,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=48,tokenizer=deepseek-ai/DeepSeek-V3.2,seed=42,max_gen_toks=12800,timeout=3600

|   Tasks   |Version|    Filter    |n-shot|  Metric   |   |Value|   |Stderr|
|-----------|------:|--------------|-----:|-----------|---|----:|---|-----:|
|engineering|    2.1|custom-extract|     5|exact_match|↑  |0.419|±  |0.0159|

Environment

root@5dc140841c7a:/sgl-workspace/sglang# python3 -m sglang.check_env
Python: 3.12.12 (main, Oct 10 2025, 08:52:57) [GCC 11.4.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA B200
GPU 0,1,2,3,4,5,6,7 Compute Capability: 10.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.9, V12.9.86
CUDA Driver Version: 580.95.05
PyTorch: 2.9.1+cu129
sglang: 0.5.6
sgl_kernel: 0.3.18.post2
flashinfer_python: 0.5.3
flashinfer_cubin: 0.5.3
flashinfer_jit_cache: Module Not Found
triton: 3.5.1
transformers: 4.57.1
torchao: 0.9.0
numpy: 2.3.5
aiohttp: 3.13.2
fastapi: 0.123.5
hf_transfer: 0.1.9
huggingface_hub: 0.36.0
interegular: 0.3.3
modelscope: 1.32.0
orjson: 3.11.4
outlines: 0.1.11
packaging: 25.0
psutil: 7.1.3
pydantic: 2.12.5
python-multipart: 0.0.20
pyzmq: 27.1.0
uvicorn: 0.38.0
uvloop: 0.22.1
vllm: Module Not Found
xgrammar: 0.1.27
openai: 2.6.1
tiktoken: 0.12.0
anthropic: 0.75.0
litellm: Module Not Found
decord2: 2.0.0
NVIDIA Topology: 
	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV18	NV18	NV18	NV18	NV18	NV18	NV18	0-55,112-167	0	N/A
GPU1	NV18	 X 	NV18	NV18	NV18	NV18	NV18	NV18	0-55,112-167	0	N/A
GPU2	NV18	NV18	 X 	NV18	NV18	NV18	NV18	NV18	0-55,112-167	0	N/A
GPU3	NV18	NV18	NV18	 X 	NV18	NV18	NV18	NV18	0-55,112-167	0	N/A
GPU4	NV18	NV18	NV18	NV18	 X 	NV18	NV18	NV18	56-111,168-223	1	N/A
GPU5	NV18	NV18	NV18	NV18	NV18	 X 	NV18	NV18	56-111,168-223	1	N/A
GPU6	NV18	NV18	NV18	NV18	NV18	NV18	 X 	NV18	56-111,168-223	1	N/A
GPU7	NV18	NV18	NV18	NV18	NV18	NV18	NV18	 X 	56-111,168-223	1	N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Hypervisor vendor:: KVM
ulimit soft: 1024
root@5dc140841c7a:/sgl-workspace/sglang# 

[Fridge003]

@pdasgup What is the official score for this dataset?
Will it be better if you disable MTP?

[pdasgup]

I modified the eval script and was able to achieve 81.27 for this model. From the paper, they were able to achieve 85.0 https://arxiv.org/pdf/2512.02556.

The more pressing issue is the flaky function call issue though #14435 (comment)

[JustinTong0323]

Are you using the thinking mode?

[YAMY1234]

Can you try using the latest SGLang code instead of sglang:v0.5.6 (you may need to pip install -e . again)? There were two NSA-related bug fixes in the past week, and I’ve confirmed that neither of them is included in the v0.5.6 tag:

• Fix NSA Bug in Centralize NSA Dispatch Logic #14245
• [DeepseekV3.2][NSA][Indexer] Fix PAGED top-k transform for NSA indexer chunked execution on H200 #14325

These bugss & fixes mainly affect H200, but it’s possible they could also impact accuracy on B200. I will also try your setting today.

[YAMY1234]

@pdasgup

I just tried to reproduce this on an 8×GB200 setup using the latest sglang code and the results look normal to me:

| Tasks       | Version | Filter         | n-shot | Metric      |   |  Value |   | Stderr |
| ----------- | ------: | -------------- | -----: | ----------- | - | -----: | - | -----: |
| engineering |     2.1 | custom-extract |      5 | exact_match | ↑ | 0.7554 | ± | 0.0138 |

Server launch command:

python -m sglang.launch_server \
  --model deepseek-ai/DeepSeek-V3.2-Exp \
  --tp 8 \
  --dp 8 \
  --enable-dp-attention

Eval command:

lm_eval --apply_chat_template \
  --model local-chat-completions \
  --tasks mmlu_pro_engineering \
  --model_args model=deepseek-ai/DeepSeek-V3.2,base_url=http://127.0.0.1:30000/v1/chat/completions,num_concurrent=48,tokenizer=deepseek-ai/DeepSeek-V3.2,seed=42,max_gen_toks=12800,timeout=3600

[b8zhong]

Yamy, I think @pdasgup is using V3.2 (non-exp)

python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.2 \
  --trust-remote-code \
  --tp-size 8 --dp-size 8 --enable-dp-attention \
  --tool-call-parser deepseekv32 \
  --reasoning-parser deepseek-v3 --port=8000 --host=0.0.0.0

lm_eval --apply_chat_template --model local-chat-completions --tasks mmlu_pro_engineering --model_args model=deepseek-ai/DeepSeek-V3.2,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=48,tokenizer=deepseek-ai/DeepSeek-V3.2,seed=42,max_gen_toks=12800,timeout=3600

|   Tasks   |Version|    Filter    |n-shot|  Metric   |   |Value |   |Stderr|
|-----------|------:|--------------|-----:|-----------|---|-----:|---|-----:|
|engineering|    2.1|custom-extract|     5|exact_match|↑  |0.4097|±  |0.0158|

Non-spec:

python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.2 \
  --trust-remote-code \
  --tp-size 8 --dp-size 8 --enable-dp-attention \
  --tool-call-parser deepseekv32 \
  --reasoning-parser deepseek-v3 --port=8000 --host=0.0.0.0

|   Tasks   |Version|    Filter    |n-shot|  Metric   |   |Value |   |Stderr|
|-----------|------:|--------------|-----:|-----------|---|-----:|---|-----:|
|engineering|    2.1|custom-extract|     5|exact_match|↑  |0.4334|±  |0.0159|

Non-spec + no chat template/parser:

python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.2 \
  --trust-remote-code \
  --tp-size 8 --dp-size 8 --enable-dp-attention \
  --port=8000 --host=0.0.0.0

|   Tasks   |Version|    Filter    |n-shot|  Metric   |   |Value |   |Stderr|
|-----------|------:|--------------|-----:|-----------|---|-----:|---|-----:|
|engineering|    2.1|custom-extract|     5|exact_match|↑  |0.4396|±  | 0.016|

With think + chat template:

python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.2 \
  --trust-remote-code \
  --tp-size 8 --dp-size 8 --enable-dp-attention \
  --tool-call-parser deepseekv32 \
  --reasoning-parser deepseek-v3 --port=8000 --host=0.0.0.0

|   Tasks   |Version|    Filter    |n-shot|  Metric   |   |Value |   |Stderr|
|-----------|------:|--------------|-----:|-----------|---|-----:|---|-----:|
|engineering|    2.1|custom-extract|     5|exact_match|↑  |0.4262|±  |0.0159|

With V3.2-Exp:

python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V3.2-Exp \
  --trust-remote-code \
  --tp-size 8 --dp-size 8 --enable-dp-attention \
  --port=8000 --host=0.0.0.0

|   Tasks   |Version|    Filter    |n-shot|  Metric   |   |Value|   |Stderr|
|-----------|------:|--------------|-----:|-----------|---|----:|---|-----:|
|engineering|    2.1|custom-extract|     5|exact_match|↑  |0.743|±  | 0.014|

Anyway, my lm-eval command might not be right to enable thinking.

[lavdnone2]

Same here with, but AWQ. Not fully hallucinating , but atrocious results. With SGLANG_ENABLE_JIT_DEEPGEMM=0 on SM120
MMLU-Pro (16 timed out)
business 176/789 wrong (77.7% accuracy)
law 624/1101 wrong (43.3% accuracy)
psychology 298/798 wrong (62.7% accuracy)
biology 176/717 wrong (75.5% accuracy)
chemistry 265/1132 wrong (76.6% accuracy)
history 168/381 wrong (55.9% accuracy)
other 319/924 wrong (65.5% accuracy)
health 267/818 wrong (67.4% accuracy)
economics 226/844 wrong (73.2% accuracy)
math 224/1351 wrong (83.4% accuracy)
physics 294/1299 wrong (77.4% accuracy)
computer science 103/410 wrong (74.9% accuracy)
philosophy 183/499 wrong (63.3% accuracy)
engineering 392/969 wrong (59.5% accuracy)

ALL CATEGORIES 3715/12032 wrong (69.1% accuracy)

Just for comparison how it should look:
QuantTrio/DeepSeek-V3.1-AWQ-Lite Nonthinking 1 timeout (0/1) (vs 83.7% full posted)
business 86/789 wrong (89.1% accuracy)
law 394/1101 wrong (64.2% accuracy)
psychology 131/798 wrong (83.6% accuracy)
biology 71/717 wrong (90.1% accuracy)
chemistry 137/1132 wrong (87.9% accuracy)
history 96/381 wrong (74.8% accuracy)
other 180/924 wrong (80.5% accuracy)
health 170/818 wrong (79.2% accuracy)
economics 109/844 wrong (87.1% accuracy)
math 101/1351 wrong (92.5% accuracy)
physics 156/1299 wrong (88.0% accuracy)
computer science 53/410 wrong (87.1% accuracy)
philosophy 102/499 wrong (79.6% accuracy)
engineering 209/969 wrong (78.4% accuracy)

ALL CATEGORIES 1995/12032 wrong (83.4% accuracy)

[lavdnone2]

Maybe related #14902

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.