refactor apply_w8a8_block_fp8_linear in fp by ChangyiYang · Pull Request #6545 · sgl-project/sglang

ChangyiYang · 2025-05-23T05:46:07Z

Motivation

refactor apply_w8a8_block_fp8_linear to make the logic more clear and more meaningful

Modifications

only refactor apply_w8a8_block_fp8_linear in fb8_utils

Checklist

Format your code according to the Code Formatting with Pre-Commit.
Add unit tests as outlined in the Running Unit Tests.
Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

python/sglang/srt/layers/quantization/fp8.py

python/sglang/srt/layers/quantization/fp8_utils.py

ChangyiYang · 2025-05-25T23:02:23Z

Hi! I have create a new commit. Here are some bullet points

Refactored dispatch logic to return only the selected matmul implementation function at initialization time to avoid filtering overhead.
For implementations requiring runtime validation (e.g., shape or dtype constraints), added fallback to Triton when conditions are not met.
Moved runtime condition checks ahead of computation to avoid redundant operations and unnecessary overhead.
Extracted w8a8_block_fp8_matmul_deepgemm from w8a8_block_fp8_matmul to enable direct usage of DeepGEMM kernels when applicable.
Retained the function w8a8_block_fp8_matmul as the unified entry point for matrix multiplication, to avoid requiring users to manually check whether DeepGEMM is available each time they use it.

Some choices I make, which may potentially have some improve space

In w8a8_block_fp8_matmul_deepgemm, use the same assertion check as w8a8_block_fp8_matmul, which may have some unnecessary check

Please check if any futhur modification is needed!

python/sglang/srt/layers/quantization/fp8_kernel.py

ChangyiYang · 2025-05-26T18:48:53Z

Hi ! Here are some modification for this commit

split out w8a8_block_fp8_matmul_triton
retain w8a8_block_fp8_matmul only for testing purpose
for benchmarks that use deepgemm, directly call w8a8_block_fp8_matmul_deepgemm

Feel free to tell me if there anything needs further modification!

sgl-kernel/benchmark/bench_fp8_blockwise_gemm.py

ChangyiYang · 2025-05-26T19:33:22Z

Hi! Typo fixed. Feel free to tell me if there is any more issue!

zhyncs · 2025-05-27T03:42:24Z

cc @HaiShaw

ChangyiYang · 2025-05-27T03:53:13Z

@Alcanderian hi, I click update branch to merge from main and seems needs to be approved again. Can you kindly approve again?

sgl-kernel/benchmark/bench_fp8_blockwise_gemm.py

Fridge003 · 2025-05-28T23:06:10Z

Hi @ChangyiYang , please fix the conflict caused by new flashinfer gemm method in #6479

Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com> Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com>

Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>

…gl-project#6703) Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>

Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>

…roject#6712)

Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>

…gl-project#6479)

ChangyiYang · 2025-05-29T04:55:46Z

Hi! Conflict solved! Add a new method flashinfer_gemm_w8a8_block_fp8_linear. Feel free to tell me if any more modification needed before merging!

Fridge003

LGTM

fzyzcjy · 2025-06-05T08:31:34Z

Hi, it seems this PR causes fused_qkv_a_proj_with_mqa to slow down 3x, thus I temporarily make a fix here #6890. I am wondering why the shape_supported is multiple of 128, do you observe bad performance or errors when it is not mult of 128?

Merge branch 'sgl_20250610_sync_tag047 of git@code.alipay.com:Theta/SGLang.git into main https://code.alipay.com/Theta/SGLang/pull_requests/52 Reviewed-by: 剑川 <jianchuan.gys@antgroup.com> * [Bugfix] Fix slice operation when chunk size mismatch (sgl-project#6697) * [Bugfix] Fix ChatCompletion endpoint of mini_lb when stream is set (sgl-project#6703) * [CI] Fix setup of disaggregation with different tp (sgl-project#6706) * [PD] Remove Unnecessary Exception Handling for FastQueue.get() (sgl-project#6712) * Fuse routed_scaling_factor in DeepSeek (sgl-project#6710) * Overlap two kernels in DeepSeek with communication (sgl-project#6711) * Minor refactor two-batch overlap (sgl-project#6682) * Speed up when having padding tokens two-batch overlap (sgl-project#6668) * [Feature] Support Flashinfer fp8 blockwise GEMM kernel on Blackwell (sgl-project#6479) * Fix LoRA bench (sgl-project#6719) * temp * Fix PP for Qwen3 MoE (sgl-project#6709) * [feat] triton kernel for get_last_loc (sgl-project#6676) * [fix] more mem for draft_extend cuda_graph (sgl-project#6726) * [PD] bug fix: Update status if nixl receiver send a a dummy req. (sgl-project#6720) * Tune memory arguments on B200 (sgl-project#6718) * Add DeepSeek-R1-0528 function call chat template (sgl-project#6725) * refactor(tool call): Fix BaseFormatDetector tool_index issue and refactor `parse_streaming_increment` (sgl-project#6715) * Add draft extend CUDA graph for Triton backend (sgl-project#6705) * refactor apply_w8a8_block_fp8_linear in fp (sgl-project#6545) * [PD] Support completion endpoint (sgl-project#6729) * PD Rust LB (PO2) (sgl-project#6437) * Super tiny enable sole usage of expert distribution metrics and update doc (sgl-project#6680) * Support picking variants of EPLB algorithms (sgl-project#6728) * Support tuning DeepEP configs (sgl-project#6742) * [test] add ut and bm for get_last_loc (sgl-project#6746) * Fix mem_fraction_static for AMD CI (sgl-project#6748) * [fix][RL] Fix DeepSeekV3ForCausalLM.post_load_weights for multiple update weight (sgl-project#6265) * Improve EPLB logical to physical dispatch map (sgl-project#6727) * Update DeepSeek-R1-0528 function call chat template (sgl-project#6765) * [PD] Optimize time out logic and add env var doc for mooncake (sgl-project#6761) * Fix aiohttp 'Chunk too big' in bench_serving (sgl-project#6737) * Support sliding window in triton backend (sgl-project#6509) * Fix shared experts fusion error (sgl-project#6289) * Fix one bug in the grouped-gemm triton kernel (sgl-project#6772) * update llama4 chat template and pythonic parser (sgl-project#6679) * feat(tool call): Enhance Llama32Detector for improved JSON parsing in non-stream (sgl-project#6784) * Support token-level quantization for EP MoE (sgl-project#6782) * Temporarily lower mmlu threshold for triton sliding window backend (sgl-project#6785) * ci: relax test_function_call_required (sgl-project#6786) * Add intel_amx backend for Radix Attention for CPU (sgl-project#6408) * Fix incorrect LoRA weight loading for fused gate_up_proj (sgl-project#6734) * fix(PD-disaggregation): Can not get local ip (sgl-project#6792) * [FIX] mmmu bench serving result display error (sgl-project#6525) (sgl-project#6791) * Bump torch to 2.7.0 (sgl-project#6788) * chore: bump sgl-kernel v0.1.5 (sgl-project#6794) * Improve profiler and integrate profiler in bench_one_batch_server (sgl-project#6787) * chore: upgrade sgl-kernel v0.1.5 (sgl-project#6795) * [Minor] Always append newline after image token when parsing chat message (sgl-project#6797) * Update CI tests for Llama4 models (sgl-project#6421) * [Feat] Enable PDL automatically on Hopper architecture (sgl-project#5981) * chore: update blackwell docker (sgl-project#6800) * misc: cache is_hopper_arch (sgl-project#6799) * Remove contiguous before Flashinfer groupwise fp8 gemm (sgl-project#6804) * Correctly abort the failed grammar requests & Improve the handling of abort (sgl-project#6803) * [EP] Add cuda kernel for moe_ep_pre_reorder (sgl-project#6699) * Add draft extend CUDA graph for flashinfer backend (sgl-project#6805) * Refactor CustomOp to avoid confusing bugs (sgl-project#5382) * Tiny log prefill time (sgl-project#6780) * Tiny fix EPLB assertion about rebalancing period and recorder window size (sgl-project#6813) * Add simple utility to dump tensors for debugging (sgl-project#6815) * Fix profiles do not have consistent names (sgl-project#6811) * Speed up rebalancing when using non-static dispatch algorithms (sgl-project#6812) * [1/2] Add Kernel support for Cutlass based Fused FP4 MoE (sgl-project#6093) * [Router] Fix k8s Service Discovery (sgl-project#6766) * Add CPU optimized kernels for topk and rope fusions (sgl-project#6456) * fix new_page_count_next_decode (sgl-project#6671) * Fix wrong weight reference in dynamic EPLB (sgl-project#6818) * Minor add metrics to expert location updater (sgl-project#6816) * [Refactor] Rename `n_share_experts_fusion` as `num_fused_shared_experts` (sgl-project#6735) * [FEAT] Add transformers backend support (sgl-project#5929) * [fix] recover auto-dispatch for rmsnorm and rope (sgl-project#6745) * fix ep_moe_reorder kernel bugs (sgl-project#6858) * [Refactor] Multimodal data processing for VLM (sgl-project#6659) * Decoder-only Scoring API (sgl-project#6460) * feat: add dp-rank to KV events (sgl-project#6852) * Set `num_fused_shared_experts` as `num_shared_experts` when shared_experts fusion is not disabled (sgl-project#6736) * Fix one missing arg in DeepEP (sgl-project#6878) * Support LoRA in TestOpenAIVisionServer and fix fused kv_proj loading bug. (sgl-project#6861) * support 1 shot allreduce in 1-node and 2-node using mscclpp (sgl-project#6277) * Fix Qwen3MoE missing token padding optimization (sgl-project#6820) * Tiny update error hints (sgl-project#6846) * Support layerwise rebalancing experts (sgl-project#6851) * Tiny allow profiler API to auto create directory (sgl-project#6865) * Support Blackwell DeepEP docker images (sgl-project#6868) * [EP] Add cuda kernel for moe_ep_post_reorder (sgl-project#6837) * [theta]merge 0605 * oai: fix openAI client error with single request via batch api (sgl-project#6170) * [PD] Fix potential perf spike caused by tracker gc and optimize doc (sgl-project#6764) * Use deepgemm instead of triton for fused_qkv_a_proj_with_mqa (sgl-project#6890) * [CUTLASS-FP4-MOE] Introduce CutlassMoEParams class for easy initialization of Cutlass Grouped Gems Metadata (sgl-project#6887) * bugfix(OAI): Fix image_data processing for jinja chat templates (sgl-project#6877) * [CPU] enable CI for PRs, add Dockerfile and auto build task (sgl-project#6458) * AITER backend extension and workload optimizations (sgl-project#6838) * [theta]merge * [theta]merge * [Feature] Support Flashinfer fmha on Blackwell (sgl-project#6930) * Fix a bug in abort & Improve docstrings for abort (sgl-project#6931) * Tiny support customize DeepEP max dispatch tokens per rank (sgl-project#6934) * Sync the changes on cuda graph runners (sgl-project#6932) * [PD] Optimize transfer queue forward logic for dummy rank (sgl-project#6922) * [Refactor] image data process in bench_serving (sgl-project#6879) * [fix] logical_to_all_physical_map index 256 is out of bounds in EP parallel. (sgl-project#6767) * Add triton fused moe kernel config for E=257 on B200 (sgl-project#6939) * [sgl-kernel] update deepgemm (sgl-project#6942) * chore: bump sgl-kernel v0.1.6 (sgl-project#6943) * Minor compile fused topk (sgl-project#6944) * [Bugfix] pipeline parallelism and Eagle Qwen2 (sgl-project#6910) * Tiny re-introduce profile id logging (sgl-project#6912) * Add triton version as a fused_moe_triton config search key to avoid performace decrease in different Triton version (sgl-project#5955) * reduce torch.zeros overhead in moe align block size kernel (sgl-project#6369) * chore: upgrade sgl-kernel v0.1.6 (sgl-project#6945) * add fbgemm moe grouped gemm kernel benchmark (sgl-project#6924) * [Docker] Add docker file for SGL Router (sgl-project#6915) * Disabling mixed chunked prefill when eagle is enabled (sgl-project#6874) * Add canary for EPLB rebalancing (sgl-project#6895) * Refactor global_server_args_dict (sgl-project#6866) * Fuse routed scaling factor in topk_reduce kernel (sgl-project#6220) * Update server timeout time in AMD CI. (sgl-project#6953) * [misc] add is_cpu() (sgl-project#6950) * Add H20 fused MoE kernel tuning configs for DeepSeek-R1/V3 (sgl-project#6885) * Add a CUDA kernel for fusing mapping and weighted sum for MoE. (sgl-project#6916) * chore: bump sgl-kernel v0.1.6.post1 (sgl-project#6955) * chore: upgrade sgl-kernel v0.1.6.post1 (sgl-project#6957) * [DeepseekR1-FP4] Add Support for nvidia/DeepSeekR1-FP4 model (sgl-project#6853) * Revert "Fuse routed scaling factor in topk_reduce kernel (sgl-project#6220)" (sgl-project#6968) * [AMD] Add more tests to per-commit-amd (sgl-project#6926) * chore: bump sgl-kernel v0.1.7 (sgl-project#6963) * Slightly improve the sampler to skip unnecessary steps (sgl-project#6956) * rebase h20 fused_moe config (sgl-project#6966) * Fix CI and triton moe Configs (sgl-project#6974) * Remove unnecessary kernels of num_token_non_padded (sgl-project#6965) * Extend cuda graph capture bs for B200 (sgl-project#6937) * Fuse routed scaling factor in deepseek (sgl-project#6970) * Sync cuda graph runners (sgl-project#6976) * Fix draft extend ut stability with flush cache (sgl-project#6979) * Fix triton sliding window test case (sgl-project#6981) * Fix expert distribution dumping causes OOM (sgl-project#6967) * Minor remove one kernel for DeepSeek (sgl-project#6977) * [perf][sgl-kernel] extend cutlass_mla_decode to support num_head < 128 (sgl-project#6929) * Enable more unit tests for AMD CI. (sgl-project#6983) * Use torch.compile to fuse flash attention decode metadata preparation (sgl-project#6973) * Eliminate stream sync to speed up LoRA batch init (sgl-project#6960) * support qwen3 emebedding (sgl-project#6990) * Fix torch profiler bugs for bench_offline_throughput.py (sgl-project#6557) * chore: upgrade flashinfer v0.2.6.post1 jit (sgl-project#6958) * cleanup tmp dir (sgl-project#7007) * chore: update pr test xeon (sgl-project#7008) * Fix cutlass MLA gets almost zero accuracy (sgl-project#6998) * Update amd nightly models CI. (sgl-project#6992) * feat: add direct routing strategy to DP worker (sgl-project#6884) * Fallback to lower triton version for unfound fused moe configs (sgl-project#7013) * Fix torchvision version for Blackwell (sgl-project#7015) * Simplify prepare_extend_after_decode (sgl-project#6987) * Migrate to assertEqual (sgl-project#6741) * Fix torch version in blackwell dockerfile (sgl-project#7017) * chore: update pr test xeon (sgl-project#7018) * Update default settings for blackwell (sgl-project#7023) * Support both approximate and exact expert distribution collection (sgl-project#6964) * Add decode req pool (sgl-project#6980) * [theta]merge 0610 * [theta]merge 0610 * [CI] Add CI workflow for sgl-router docker build (sgl-project#7027) * Fix fused_moe triton configs (sgl-project#7029) * CPU: map changes from developing branch in sgl-kernel (sgl-project#6833) * chore: bump v0.4.7 (sgl-project#7038) * Update README.md (sgl-project#7040)

ChangyiYang requested review from BBuf, HaiShaw, Ying1123, ch-wan, ispobock, merrymercy and zhyncs as code owners May 23, 2025 05:46

ChangyiYang mentioned this pull request May 23, 2025

[Feature] add abstraction for different platform #4353

Closed

2 tasks

Fridge003 reviewed May 23, 2025

View reviewed changes

python/sglang/srt/layers/quantization/fp8.py Outdated Show resolved Hide resolved

Alcanderian reviewed May 23, 2025

View reviewed changes

python/sglang/srt/layers/quantization/fp8_utils.py Outdated Show resolved Hide resolved

Fridge003 reviewed May 26, 2025

View reviewed changes

python/sglang/srt/layers/quantization/fp8_kernel.py Outdated Show resolved Hide resolved

ChangyiYang requested a review from Alcanderian May 26, 2025 04:38

zhyncs assigned Alcanderian, BBuf and Fridge003 May 26, 2025

BBuf reviewed May 26, 2025

View reviewed changes

python/sglang/srt/layers/quantization/fp8_kernel.py Outdated Show resolved Hide resolved

ChangyiYang requested review from FlamingoPg, HandH1998 and yizhang2077 as code owners May 26, 2025 18:46

Alcanderian reviewed May 26, 2025

View reviewed changes

sgl-kernel/benchmark/bench_fp8_blockwise_gemm.py Outdated Show resolved Hide resolved

ChangyiYang requested a review from BBuf May 27, 2025 03:09

Alcanderian approved these changes May 27, 2025

View reviewed changes

zhyncs assigned HaiShaw May 27, 2025

Alcanderian reviewed May 27, 2025

View reviewed changes

sgl-kernel/benchmark/bench_fp8_blockwise_gemm.py Outdated Show resolved Hide resolved

refactor apply_w8a8_block_fp8_linear

47a5c62

whybeyoung and others added 11 commits May 28, 2025 21:25

[PD Perf] replace Queue to FastQueue (sgl-project#6649)

bdab9a1

Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com> Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com>

[Bugfix] Fix slice operation when chunk size mismatch (sgl-project#6697)

279885a

Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>

[Bugfix] Fix ChatCompletion endpoint of mini_lb when stream is set (s…

8f84f9c

…gl-project#6703) Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>

[CI] Fix setup of disaggregation with different tp (sgl-project#6706)

0de019b

Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>

[PD] Remove Unnecessary Exception Handling for FastQueue.get() (sgl-p…

cbf1e96

…roject#6712)

Fuse routed_scaling_factor in DeepSeek (sgl-project#6710)

1a55a95

Overlap two kernels in DeepSeek with communication (sgl-project#6711)

6374a27

Minor refactor two-batch overlap (sgl-project#6682)

cdedbb3

Speed up when having padding tokens two-batch overlap (sgl-project#6668)

93ef744

Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>

[Feature] Support Flashinfer fp8 blockwise GEMM kernel on Blackwell (s…

7df4699

…gl-project#6479)

create new dispatching function flashinfer_gemm_w8a8_block_fp8_linear

8368a5d

ChangyiYang requested review from ByronHsu, hnyls2002 and zhaochenyang20 as code owners May 29, 2025 04:41

Merge branch 'main' into refactor_apply_w8a8_block_fp8_linear

2664692

ChangyiYang added 2 commits May 28, 2025 23:51

Merge branch 'main' into refactor_apply_w8a8_block_fp8_linear

4ef9146

Merge branch 'main' into refactor_apply_w8a8_block_fp8_linear

37bf9c9

ChangyiYang requested a review from Fridge003 May 29, 2025 07:04

Fridge003 approved these changes May 29, 2025

View reviewed changes

zhyncs approved these changes May 29, 2025

View reviewed changes

Merge branch 'main' into refactor_apply_w8a8_block_fp8_linear

d4e3cf6

zhyncs merged commit 485a023 into sgl-project:main May 29, 2025

ChangyiYang mentioned this pull request Jun 1, 2025

Refactor apply fp8 linear #6802

Open

6 tasks

Layssy pushed a commit to Layssy/sglang-iaas that referenced this pull request Jun 9, 2025

refactor apply_w8a8_block_fp8_linear in fp (sgl-project#6545)

a90ece5

xwu-intel pushed a commit to xwu-intel/sglang that referenced this pull request Jun 17, 2025

refactor apply_w8a8_block_fp8_linear in fp (sgl-project#6545)

fa0cae8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor apply_w8a8_block_fp8_linear in fp#6545

refactor apply_w8a8_block_fp8_linear in fp#6545
zhyncs merged 26 commits intosgl-project:mainfrom
ChangyiYang:refactor_apply_w8a8_block_fp8_linear

ChangyiYang commented May 23, 2025

Uh oh!

Uh oh!

Uh oh!

ChangyiYang commented May 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

ChangyiYang commented May 26, 2025

Uh oh!

Uh oh!

ChangyiYang commented May 26, 2025

Uh oh!

zhyncs commented May 27, 2025

Uh oh!

ChangyiYang commented May 27, 2025

Uh oh!

Uh oh!

Fridge003 commented May 28, 2025

Uh oh!

ChangyiYang commented May 29, 2025 •

edited

Loading

Uh oh!

Fridge003 left a comment

Uh oh!

fzyzcjy commented Jun 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

Conversation

ChangyiYang commented May 23, 2025

Motivation

Modifications

Checklist

Uh oh!

Uh oh!

Uh oh!

ChangyiYang commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChangyiYang commented May 26, 2025

Uh oh!

Uh oh!

ChangyiYang commented May 26, 2025

Uh oh!

zhyncs commented May 27, 2025

Uh oh!

ChangyiYang commented May 27, 2025

Uh oh!

Uh oh!

Fridge003 commented May 28, 2025

Uh oh!

ChangyiYang commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fridge003 left a comment

Choose a reason for hiding this comment

Uh oh!

fzyzcjy commented Jun 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

ChangyiYang commented May 25, 2025 •

edited

Loading

ChangyiYang commented May 29, 2025 •

edited

Loading