use flashinfer_trtllm moe runner backend to gain around 10% perf on b200 fp8 dpsk by b8zhong · Pull Request #11816 · sgl-project/sglang

b8zhong · 2025-10-19T01:25:42Z

Motivation

Do 3 things

If on SM100 + FP8 for dpsk, use flashinfer_trtllm as default moe runner backend, replacing triton (current default). the fp4 path is already using this. Note that shared experts fusion is disabled here.

The performance improvements are described:

https://sgl-fru7574.slack.com/archives/C0999LZPKQX/p1760830685045249

add the tuned triton config, there was around a 5% improvement in the overall kernel itself, but the E2E results above (those are with triton tuned results) are still worse.
Remove SGLANG_CUTLASS_MOE and move it to server args (like all other moe runner backends.

python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1 --tp 8

Launches with FI trtllm moe runner now.

python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1 --tp 8 --moe-runner-backend cutlass

Also works now.

Accuracy Tests

python3 benchmark/gsm8k/bench_sglang.py --num-shots 8 --num-questions 1319 --parallel 1319
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [00:25<00:00, 51.73it/s]
Accuracy: 0.952
Invalid: 0.001
Latency: 25.638 s
Output throughput: 5178.826 token/s

gemini-code-assist · 2025-10-19T01:26:02Z

Summary of Changes

Hello @b8zhong, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily focuses on enhancing the performance of Mixture-of-Experts (MoE) models, particularly for FP8 quantization on NVIDIA Blackwell (SM100/B200) GPUs. It achieves this by making flashinfer_trtllm the default MoE runner backend, which has demonstrated approximately 10% performance improvement. Additionally, it introduces optimized Triton configurations and refactors the MoE kernel selection mechanism for improved maintainability and configurability, allowing users to specify MoE backends directly through server arguments.

Highlights

MoE Runner Backend Update: The default MoE runner backend for FP8 quantization on SM100 (Blackwell) GPUs is now flashinfer_trtllm, replacing Triton, to achieve significant performance gains (around 10% on B200).
Tuned Triton Configuration: A new tuned Triton configuration has been added for FP8 MoE on NVIDIA B200 GPUs, offering kernel-level improvements, although flashinfer_trtllm still yields better end-to-end results.
Refactored MoE Kernel Selection: The logic for selecting and initializing Cutlass FP8 fused MoE kernels has been refactored, moving the SGLANG_CUTLASS_MOE environment variable to be managed via server arguments and encapsulating the selection criteria and buffer initialization into dedicated methods.
Environment Variable Removal: The SGLANG_CUTLASS_MOE environment variable has been removed from both the codebase and documentation, streamlining configuration.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces performance improvements for MoE models on B200 GPUs by defaulting to the flashinfer_trtllm backend for FP8 quantization. It also refactors the MoE backend selection by removing the SGLANG_CUTLASS_MOE environment variable in favor of a server argument, which is a good cleanup. Additionally, a new tuned Triton config for B200 is included. I've found a critical issue in the logic for setting the default MoE runner backend due to operator precedence, which could lead to incorrect behavior. Please see the detailed comment.

python/sglang/srt/server_args.py

Qiaolin-Yu

could you share the moe kernel profiling result before and after tuning?

b8zhong · 2025-10-19T03:21:10Z

@Qiaolin-Yu I didn't use torch profiler, but I generally find the benchmark provided by this tuning script to be accurate

python3 benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton.py --model deepseek-ai/DeepSeek-R1 --tp 8 --dtype w8a8
# then use --tune for tuning, omit to get benchmark results

Before:

Batch size: 1, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 52.65 us
Batch size: 2, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 64, 'num_warps': 4, 'num_stages': 4}
Kernel time: 60.35 us
Batch size: 4, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 64, 'num_warps': 4, 'num_stages': 4}
Kernel time: 79.09 us
Batch size: 8, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 95.47 us
Batch size: 16, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 138.68 us
Batch size: 24, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 16, 'num_warps': 4, 'num_stages': 3}
Kernel time: 175.80 us
Batch size: 32, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 64, 'GROUP_SIZE_M': 16, 'num_warps': 4, 'num_stages': 3}
Kernel time: 215.57 us
Batch size: 48, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 64, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 238.00 us
Batch size: 64, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 262.97 us
Batch size: 96, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 64, 'num_warps': 4, 'num_stages': 3}
Kernel time: 290.94 us
Batch size: 128, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 64, 'num_warps': 4, 'num_stages': 3}
Kernel time: 298.54 us
Batch size: 256, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 16, 'num_warps': 4, 'num_stages': 3}
Kernel time: 313.66 us
Batch size: 512, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 16, 'num_warps': 4, 'num_stages': 3}
Kernel time: 404.51 us
Batch size: 1024, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 16, 'num_warps': 8, 'num_stages': 4}
Kernel time: 601.92 us
Batch size: 1536, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 8, 'num_stages': 4}
Kernel time: 659.24 us
Batch size: 2048, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 16, 'num_warps': 8, 'num_stages': 4}
Kernel time: 862.54 us
Batch size: 3072, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 8, 'num_stages': 4}
Kernel time: 1131.79 us
Batch size: 4096, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 8, 'num_stages': 4}
Kernel time: 1412.09 us

After:

Batch size: 1, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 52.67 us
Batch size: 2, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 59.37 us
Batch size: 4, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 77.77 us
Batch size: 8, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 95.52 us
Batch size: 16, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 138.66 us
Batch size: 24, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 169.08 us
Batch size: 32, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 64, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 210.91 us
Batch size: 48, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 64, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 237.98 us
Batch size: 64, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 258.92 us
Batch size: 96, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 277.91 us
Batch size: 128, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 286.46 us
Batch size: 256, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 301.79 us
Batch size: 512, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 388.22 us
Batch size: 1024, config: {'BLOCK_SIZE_M': 32, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 568.84 us
Batch size: 1536, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 4, 'num_stages': 4}
Kernel time: 642.70 us
Batch size: 2048, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 4, 'num_stages': 4}
Kernel time: 829.11 us
Batch size: 3072, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 1083.93 us
Batch size: 4096, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 4, 'num_stages': 3}
Kernel time: 1350.88 us

On average the triton config is like 2% improvement

Qiaolin-Yu · 2025-10-19T03:23:08Z

@Qiaolin-Yu I didn't use torch profiler, but I generally find the benchmark provided by this tuning script to be accurate

python3 benchmark/kernels/fused_moe_triton/tuning_fused_moe_triton.py --model deepseek-ai/DeepSeek-R1 --tp 8 --dtype w8a8
# then use --tune for tuning, omit to get benchmark results

Before:

Batch size: 1, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 52.65 us
Batch size: 2, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 64, 'num_warps': 4, 'num_stages': 4}
Kernel time: 60.35 us
Batch size: 4, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 64, 'num_warps': 4, 'num_stages': 4}
Kernel time: 79.09 us
Batch size: 8, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 95.47 us
Batch size: 16, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 138.68 us
Batch size: 24, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 16, 'num_warps': 4, 'num_stages': 3}
Kernel time: 175.80 us
Batch size: 32, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 64, 'GROUP_SIZE_M': 16, 'num_warps': 4, 'num_stages': 3}
Kernel time: 215.57 us
Batch size: 48, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 64, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 238.00 us
Batch size: 64, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 262.97 us
Batch size: 96, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 64, 'num_warps': 4, 'num_stages': 3}
Kernel time: 290.94 us
Batch size: 128, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 64, 'num_warps': 4, 'num_stages': 3}
Kernel time: 298.54 us
Batch size: 256, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 16, 'num_warps': 4, 'num_stages': 3}
Kernel time: 313.66 us
Batch size: 512, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 16, 'num_warps': 4, 'num_stages': 3}
Kernel time: 404.51 us
Batch size: 1024, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 16, 'num_warps': 8, 'num_stages': 4}
Kernel time: 601.92 us
Batch size: 1536, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 8, 'num_stages': 4}
Kernel time: 659.24 us
Batch size: 2048, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 16, 'num_warps': 8, 'num_stages': 4}
Kernel time: 862.54 us
Batch size: 3072, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 8, 'num_stages': 4}
Kernel time: 1131.79 us
Batch size: 4096, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 8, 'num_stages': 4}
Kernel time: 1412.09 us

After:

Batch size: 1, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 52.67 us
Batch size: 2, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 59.37 us
Batch size: 4, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 77.77 us
Batch size: 8, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 95.52 us
Batch size: 16, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 138.66 us
Batch size: 24, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 169.08 us
Batch size: 32, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 64, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 210.91 us
Batch size: 48, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 64, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 237.98 us
Batch size: 64, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 258.92 us
Batch size: 96, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 277.91 us
Batch size: 128, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 286.46 us
Batch size: 256, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 301.79 us
Batch size: 512, config: {'BLOCK_SIZE_M': 16, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 388.22 us
Batch size: 1024, config: {'BLOCK_SIZE_M': 32, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 3}
Kernel time: 568.84 us
Batch size: 1536, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 4, 'num_stages': 4}
Kernel time: 642.70 us
Batch size: 2048, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 4, 'num_stages': 4}
Kernel time: 829.11 us
Batch size: 3072, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 1, 'num_warps': 4, 'num_stages': 4}
Kernel time: 1083.93 us
Batch size: 4096, config: {'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 128, 'GROUP_SIZE_M': 32, 'num_warps': 4, 'num_stages': 3}
Kernel time: 1350.88 us

On average the triton config is like 2% improvement

I think it will be better to have a profiling result.

b8zhong · 2025-10-19T04:18:59Z

@Qiaolin-Yu Sure, btw it seems like bench one batch has some issues... so I profiled the server
btw, the trace always gets too big to open for me, so i only test with bs = 2 on server
(very small diff like he benchmark result by the script)
be

fore

after

python/sglang/srt/environ.py

.../triton_3_4_0/E=257,N=256,device_name=NVIDIA_B200,dtype=fp8_w8a8,block_shape=[128, 128].json

Qiaolin-Yu · 2025-10-22T03:30:39Z

python/sglang/srt/layers/quantization/fp8.py

+                and (is_sm100_supported() or is_sm90_supported())
+            )
+        return (
+            (backend.is_cutlass() or backend.is_flashinfer_cutlass())


i think cutlass and flashinfer_cutlass are different kernels?

And could you refine the if/else logic here? It seems a little bit messy.

Sg

ah you are right, I was not aware of pure cutlass impl... I kept is flashinfer cutlass only

Yes, I will assume the user knows the compatibility (and just return immediately true) else use the check

Qiaolin-Yu · 2025-10-22T06:26:40Z

python/sglang/srt/layers/quantization/fp8.py

                return StandardCombineInput(hidden_states=ret)

-        if self.use_cutlass_fused_experts_fp8:
+        if self._should_use_cutlass_fused_experts():


Why we move this out of if self.block_quant:?

Smth weird I did during refactor... changed it back

more

a47d67a

b8zhong requested review from BBuf, Edwardf0t1, HaiShaw, Ying1123, ch-wan, ispobock, kushanam, merrymercy and zhyncs as code owners October 19, 2025 01:25

gemini-code-assist bot reviewed Oct 19, 2025

View reviewed changes

python/sglang/srt/server_args.py Outdated Show resolved Hide resolved

more

a4ac14b

b8zhong added the run-ci label Oct 19, 2025

Qiaolin-Yu reviewed Oct 19, 2025

View reviewed changes

Fridge003 assigned Qiaolin-Yu Oct 20, 2025

Qiaolin-Yu reviewed Oct 20, 2025

View reviewed changes

python/sglang/srt/environ.py Show resolved Hide resolved

Fridge003 reviewed Oct 20, 2025

View reviewed changes

.../triton_3_4_0/E=257,N=256,device_name=NVIDIA_B200,dtype=fp8_w8a8,block_shape=[128, 128].json Outdated Show resolved Hide resolved

more

4176bb1

b8zhong requested a review from FlamingoPg as a code owner October 20, 2025 04:34

b8zhong added 2 commits October 19, 2025 21:35

more

faf2b20

Merge branch 'main' into use-trtllm-backend-default

d958d21

b8zhong requested review from Fridge003 and Qiaolin-Yu October 20, 2025 20:08

Qiaolin-Yu reviewed Oct 22, 2025

View reviewed changes

b8zhong added 2 commits October 21, 2025 23:13

more

842730b

more

a2d1d05

Qiaolin-Yu reviewed Oct 22, 2025

View reviewed changes

more

a2f7e1e

Qiaolin-Yu approved these changes Oct 22, 2025

View reviewed changes

Fridge003 approved these changes Oct 23, 2025

View reviewed changes

Fridge003 added the ready-to-merge The PR is ready to merge after the CI is green. label Oct 23, 2025

b8zhong force-pushed the use-trtllm-backend-default branch from 584a1d6 to a2f7e1e Compare October 23, 2025 18:13

Fridge003 merged commit f80371f into sgl-project:main Oct 24, 2025
201 of 271 checks passed

b8zhong deleted the use-trtllm-backend-default branch October 24, 2025 02:19

Conversation

b8zhong commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Accuracy Tests

Uh oh!

gemini-code-assist bot commented Oct 19, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Qiaolin-Yu left a comment

Choose a reason for hiding this comment

Uh oh!

b8zhong commented Oct 19, 2025

Uh oh!

Qiaolin-Yu commented Oct 19, 2025

Uh oh!

b8zhong commented Oct 19, 2025

Uh oh!

Uh oh!

Uh oh!

Qiaolin-Yu Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Qiaolin-Yu Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

b8zhong Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Qiaolin-Yu Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

b8zhong Oct 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

b8zhong commented Oct 19, 2025 •

edited

Loading