Pr/4844 by thyecust · Pull Request #1 · thyecust/sglang

thyecust · 2025-04-11T09:05:36Z

Motivation

Modifications

Checklist

Format your code according to the Code Formatting with Pre-Commit.
Add unit tests as outlined in the Running Unit Tests.
Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

…-project#4882)

…#4530)

Co-authored-by: Sleepcoo <Sleepcoo@gmail.com>

…gl-project#4932)

Co-authored-by: ch-wan <cwan39@gatech.edu>

Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>

…roject#4953)

…ject#5207)

…3/R1 (sgl-project#5196)

…ject#5086)

Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com> Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com> Co-authored-by: Xuchun Shang <xuchun.shang@linux.alibaba.com> Co-authored-by: shangmingc <csmthu@gmail.com>

…-project#5204)

… dependency (sgl-project#5245)

Co-authored-by: zhyncs <me@zhyncs.com>

…gl-project#5228)

…sgl-project#5262) Co-authored-by: ch-wan <cwan39@gatech.edu>

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

fzyzcjy and others added 30 commits April 11, 2025 17:06

Fix BadRequestError wrong arguments and remove openai dependency (sgl…

3edfd88

…-project#4882)

Improve stack trace of retry errors (sgl-project#4845)

e34ccea

Tiny fix doc error (sgl-project#4795)

409d470

[Docs] Update DeepGEMM at README.md (sgl-project#4886)

167d22c

Update CODEOWNERS (sgl-project#4889)

7d8a876

Delete test_deep_gemm.py (sgl-project#4891)

4f9d2ba

Add deepseek style fused moe group gate selection kernel (sgl-project…

5634a3a

…#4530)

quick fix: add default for new kernel (sgl-project#4898)

e6d3ba6

remove setup for sgl-kernel (sgl-project#4899)

8cca62b

[Misc] Clean m.def and add Development Tips (sgl-project#4890)

4515581

fix allreduce test (sgl-project#4909)

116c00c

Support page size > 1 + eagle (sgl-project#4908)

ac3d99a

Fix retract for page size > 1 (sgl-project#4914)

410d637

[Feature] use pytest for sgl-kernel (sgl-project#4896)

6024dff

fix bmm fp8 (sgl-project#4926)

53245c2

Fix the timeout for unit-test-2-gpu in pr-test.yml (sgl-project#4927)

a67c15e

Fix 2-gpu CI test and suppress some warnings (sgl-project#4930)

e0bba86

[feat] add fa3 in sgl-kernel (sgl-project#4902)

2c09126

Co-authored-by: Sleepcoo <Sleepcoo@gmail.com>

Fix sglang frontend's incorrect dependency on torch (sgl-project#4931)

a1630d6

[Fix] avoid stream sync and torch compile in prefill for fa3 backend (s…

1b4768a

…gl-project#4932)

cleanup sgl-kernel (sgl-project#4933)

5ff9059

[Fix] Improve Lora tests and reduce CI runtime (sgl-project#4925)

a639a05

Fix DeepSeek bug causing 2.2% MMLU drop when TP!=DP (sgl-project#4883)

8339e22

Co-authored-by: ch-wan <cwan39@gatech.edu>

[Fix] Add torch compile for torch.clamp back (sgl-project#4936)

acc9ae6

Fix oom error for large page size (sgl-project#4913)

0259f4e

Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>

[feat] interface for platforms abstraction (sgl-project#4928)

5d61e95

[Fix] revert clean m.def for cudagraph (sgl-project#4944)

c34051c

refactor: multimodal data (sgl-project#4754)

ba2b8e4

bump sgl-kernel v0.0.6 (sgl-project#4950)

151b8da

[Build] Fix cuda12.8 build error in nvfp4_scaled_mm_kernels.cu (sgl-p…

e999662

…roject#4953)

BBuf and others added 27 commits April 11, 2025 17:06

[ci] fix ci test fused_moe op (sgl-project#5102)

facf837

model: support mllama4 (sgl-project#5144)

97e80a4

update grok test (sgl-project#5171)

aae6996

sgl-kernel use cutlass latest version for fp8 blockwise gemm (sgl-pro…

77e9549

…ject#5207)

Add H20 dtype fp8_w8a8 fused MoE kernel tuning configs for DeepSeek V…

3617f1e

…3/R1 (sgl-project#5196)

fix: log warning when disable cuda graph (sgl-project#5209)

f241390

[metrics] Add in queue metrics (sgl-project#4444)

414a840

Fix DeepSeek error when using DeepEP mode (sgl-project#5190)

6fbe8d1

reduce moe_align_block_size_kernel small batch mode overhead (sgl-pro…

720818a

…ject#5086)

[PD] Add get_contiguous_buf_infos interface for MLATokenToKVPool (sgl…

a85f762

…-project#5204)

Update deps for mllama4 (sgl-project#5215)

41e1bee

Fix deepseek-v3 with torch.compile in PyTorch 2.6. (sgl-project#5213)

c097f41

ROCm sgl-kernel: compatible to later torch (sgl-project#5167)

973c449

[Misc] Clean sgl-kernel test (sgl-project#5216)

1dc3b18

Update Makefile / build script to avoid installing incompatible torch…

9684112

… dependency (sgl-project#5245)

Fix torch.compile cacheing (sgl-project#5259)

9e8a69f

Co-authored-by: zhyncs <me@zhyncs.com>

ROCm/AITER CK_MoE: update 2-stage kernels & support both Activations (s…

02f9a5e

…gl-project#5228)

Optimize attention in llama4 (sgl-project#5127)

74eb12c

Optimize GPU memory usage in FlashAttentionBackend's strided indexing (…

d5df05a

…sgl-project#5262) Co-authored-by: ch-wan <cwan39@gatech.edu>

Support --enable-llama4-multimodal (sgl-project#5254)

1d65a62

[fix] fix mrope positions not picked up (sgl-project#5265)

c9180cc

doc: nested loop code for offline engine (sgl-project#5244)

610da05

fix: examples for token_in_token_out_vlm (sgl-project#5193)

278d4d2

Fix a 404 link in send_request.ipynb (sgl-project#5280)

cfcc692

Signed-off-by: windsonsea <haifeng.yao@daocloud.io>

fix: enable fp4 compilation on cu128 (sgl-project#5286)

9371d0c

update

3e3276e

thyecust force-pushed the pr/4844 branch from 9d9e64f to 3e3276e Compare April 11, 2025 09:06

thyecust merged this pull request into fix-structured-ipynb Apr 11, 2025
1 check failed

thyecust deleted the pr/4844 branch April 11, 2025 09:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Pr/4844#1

Pr/4844#1
thyecust merged 142 commits intofix-structured-ipynbfrom
pr/4844

thyecust commented Apr 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Comments

Conversation

thyecust commented Apr 11, 2025

Motivation

Modifications

Checklist

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants