[Model] Adding Qwen3 and Qwen3MoE

[yhyang201]

This PR provides initial support for the Qwen3 and Qwen3MoE. Once the model file becomes available, I will further refine the integration.

ref huggingface/transformers#36878, vllm-project/vllm#15289

Motivation

#4682

Modifications

[yhyang201]

There are a few important points to note regarding sliding_window:

1. I noticed that in Transformers, Qwen2Attention/Qwen3Attention (from qwen2/qwen3) enable sliding_window based on the configuration. However, qwen2 in SGLang does not, so I have not introduced sliding_window in qwen3.py either.

2. Likewise, the Qwen3MoeAttention that I implemented in qwen3_moe.py does not enable sliding_window, keeping it consistent with qwen2_moe.py.

[HaiShaw]

@yhyang201

• Is pipeline parallel necessary (I see some reference to it in [Model] Add Qwen3 and Qwen3MoE vllm-project/vllm#15289)?
• What is the model weights and activation quantization scheme? FP8? Per-tensor, per channel or per block, both static or dynamic?

[HaiShaw]

Can we have dummy weight and dummy weight loading work for this too? Thanks.

[yhyang201]

Can we have dummy weight and dummy weight loading work for this too? Thanks.

1. I don’t think pipeline parallelism is necessary .
2. As far as I know, the Qwen team hasn’t released detailed information on this yet (it might be similar to Qwen2, but that’s just a guess).
3. With this fork of transformers: https://github.com/bozheng-hit/transformers/tree/qwen3, the following dummy weight loading commands work as expected:

python -m sglang.bench_one_batch --model-path qwen3 --load-format dummy 
python -m sglang.bench_one_batch --model-path qwen3moe --load-format dummy

Both models run successfully with dummy weights.

[ZhangJianwei0311]

There is a missing line k = k_by_head.view(k.shape) between lines 232 and 233 in the _apply_qk_norm method of the Attention class, qwen3_moe.py. hhh

[yhyang201]

There is a missing line k = k_by_head.view(k.shape) between lines 232 and 233 in the _apply_qk_norm method of the Attention class, qwen3_moe.py. hhh

thanks.

[zhaochenyang20]

rebase with main? Also, I added you to sglang-qwen team.

[zhaochenyang20]

Great job!

[zhaochenyang20]

1. Profiling the speed;
2. benchmarking the performance;

[yizhang2077]

LGTM overall, vllm looks like have merged qwen3/qwen3moe, could we compare the speed with it?

[yhyang201]

LGTM overall, vllm looks like have merged qwen3/qwen3moe, could we compare the speed with it?

Sure, shall we then proceed with the comparison using the parameter settings provided in https://github.com/sgl-project/sglang/tree/main/benchmark/benchmark_vllm_060?

[yizhang2077]

LGTM overall, vllm looks like have merged qwen3/qwen3moe, could we compare the speed with it?

Sure, shall we then proceed with the comparison using the parameter settings provided in https://github.com/sgl-project/sglang/tree/main/benchmark/benchmark_vllm_060?

Ok, it seems vllm has supported deepgemm in fusedmoe vllm-project/vllm#13932, I am curious about performance for vllm when turning on/off deepgemm option in qwen3moe (vllm disables deepgemm in fusedmoe by default, using VLLM_USE_DEEP_GEMM env to turn on)

[yhyang201]

The accuracy test looks good. Shall we merge? @zhaochenyang20 @zhyncs

[zhaochenyang20]

great work!

[yhyang201]

	math_opensource/aime24	math_opensource/aime25	livecodebench	ifeval, Strict_prompt_acc
qwen3-moe-235B	0.8536458333333333	0.8171875	0.660394265232975	0.8428835489833642
Qwen3-30B-A3B	0.80625
qwen3-32b	82.2

Anyone can add a link at https://docs.google.com/document/d/1gOix2oinY0UiS0qSTDoxRpxPo_klIq7cuxZMoA8vldc/edit?usp=sharing