Skip to content

enable DeepSeek V3 shared_experts_fusion in sm90 #5571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merrymercy merged 3 commits into from
Apr 20, 2025
Merged

enable DeepSeek V3 shared_experts_fusion in sm90 #5571

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f296777
upd
BBuf Apr 20, 2025
f9c55f7
lint
BBuf Apr 20, 2025
b84cd9c
Merge branch 'main' into enable_deepseek_shared_expert_fusion_in_sm90
BBuf Apr 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions python/sglang/srt/models/deepseek_v2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1427,6 +1427,18 @@ def __init__(
assert (
self.n_share_experts_fusion == self.tp_size
), f"Shared experts fusion optimization is enabled in DeepSeek V3/R1, set it to {self.tp_size} can get best optimized performace."
elif self.n_share_experts_fusion == 0:
if (
torch.cuda.get_device_capability("cuda") >= (9, 0)
and self.config.architectures[0] == "DeepseekV3ForCausalLM"
and self.config.n_routed_experts == 256
and (not global_server_args_dict["enable_deepep_moe"])
):
self.n_share_experts_fusion = self.tp_size
global_server_args_dict["n_share_experts_fusion"] = self.tp_size
logger.info(
"Deepseek V3/R1 with fp8 can use shared experts fusion optimization when SM version >=90. Shared experts fusion optimization is enabled."
)

self.model = DeepseekV2Model(
config, quant_config, prefix=add_prefix("model", prefix)
Expand Down
Loading