Pr/4844#1

Merged

thyecust merged 142 commits intofix-structured-ipynbfrom

pr/4844

Apr 11, 2025

Commits on Apr 11, 2025

Fix BadRequestError wrong arguments and remove openai dependency (#4882)

fzyzcjy
authored and
thyecust
committed
Improve stack trace of retry errors (#4845)

fzyzcjy
authored and
thyecust
committed
Tiny fix doc error (#4795)

fzyzcjy
authored and
thyecust
committed
[Docs] Update DeepGEMM at README.md (#4886)

FlamingoPg
authored and
thyecust
committed
Update CODEOWNERS (#4889)

zhyncs
authored and
thyecust
committed
Delete test_deep_gemm.py (#4891)

FlamingoPg
authored and
thyecust
committed
Add deepseek style fused moe group gate selection kernel (#4530)

qingquansong
authored and
thyecust
committed
quick fix: add default for new kernel (#4898)

FlamingoPg
authored and
thyecust
committed
remove setup for sgl-kernel (#4899)

zhyncs
authored and
thyecust
committed
[Misc] Clean m.def and add Development Tips (#4890)

FlamingoPg
authored and
thyecust
committed
fix allreduce test (#4909)

yizhang2077
authored and
thyecust
committed
Support page size > 1 + eagle (#4908)

merrymercy
authored and
thyecust
committed
Fix retract for page size > 1 (#4914)

merrymercy
authored and
thyecust
committed
[Feature] use pytest for sgl-kernel (#4896)

adarshxs
authored and
thyecust
committed
fix bmm fp8 (#4926)

zhyncs
authored and
thyecust
committed
Fix the timeout for unit-test-2-gpu in pr-test.yml (#4927)

merrymercy
authored and
thyecust
committed
Fix 2-gpu CI test and suppress some warnings (#4930)

merrymercy
authored and
thyecust
committed
[feat] add fa3 in sgl-kernel (#4902)

authored and
thyecust
committed
Fix sglang frontend's incorrect dependency on torch (#4931)

seplos
authored and
thyecust
committed
[Fix] avoid stream sync and torch compile in prefill for fa3 backend (#4932)

Fridge003
authored and
thyecust
committed
cleanup sgl-kernel (#4933)

zhyncs
authored and
thyecust
committed
[Fix] Improve Lora tests and reduce CI runtime (#4925)

Fridge003
authored and
thyecust
committed
Fix DeepSeek bug causing 2.2% MMLU drop when TP!=DP (#4883)

authored and
thyecust
committed
[Fix] Add torch compile for torch.clamp back (#4936)

Fridge003
authored and
thyecust
committed
Fix oom error for large page size (#4913)

authored and
thyecust
committed
[feat] interface for platforms abstraction (#4928)

Alcanderian
authored and
thyecust
committed
[Fix] revert clean m.def for cudagraph (#4944)

FlamingoPg
authored and
thyecust
committed
refactor: multimodal data (#4754)

mickqian
authored and
thyecust
committed
bump sgl-kernel v0.0.6 (#4950)

zhyncs
authored and
thyecust
committed
[Build] Fix cuda12.8 build error in nvfp4_scaled_mm_kernels.cu (#4953)

guoyuhong
authored and
thyecust
committed
use fa3 in sgl-kernel (#4954)

zhyncs
authored and
thyecust
committed
Revert PR 4764 & 4813 related to R1 RoPE (#4959)

guoyuhong
authored and
thyecust
committed
[Feature] Support DeepEP Low Latency (#4767)

authored and
thyecust
committed
update bench_serving (#4958)

zhyncs
authored and
thyecust
committed
Prevent memory leak of retract_decode when page_size > 1 (#4977)

xiezhq-hermann
authored and
thyecust
committed
[VLM RLHF] Take Image input for verl vlm rollout (#4915)

authored and
thyecust
committed
Large page size aligned hierarchical caching (#4581)

xiezhq-hermann
authored and
thyecust
committed
bug fix for hicache host eviction (#4989)

xiezhq-hermann
authored and
thyecust
committed
sgl scaled_fp8_quant support output padding (#4861)

BBuf
authored and
thyecust
committed
Add Eagle Speculative Decoding to FA3 Backend (#4951)

authored and
thyecust
committed
Update tokenizer_manager.py (#5008)

yangky11
authored and
thyecust
committed
[sgl-kernel] per token group quant support COLUMN MAJOR (#4817)

BBuf
authored and
thyecust
committed
update cutlass tag (#5011)

xiezhq-hermann
authored and
thyecust
committed
Feature/revise docs ci (#5009)

renxinx
authored and
thyecust
committed
fix: fix illegal cuda memory access at fused_moe_kernel (#4727)

authored and
thyecust
committed
[Build] Support build sgl-kernel with ccache (#5020)

guoyuhong
authored and
thyecust
committed
fix deepgemm as well (#5030)

xiezhq-hermann
authored and
thyecust
committed
try to fix ci oserror (#5024)

BBuf
authored and
thyecust
committed
Replace enable_flashinfer_mla argument with attention_backend (#5005)

Fridge003
authored and
thyecust
committed
Small refactor DeepEPMode to clean up code a bit (#4992)

fzyzcjy
authored and
thyecust
committed
[Fix] fix fa3 build at cu118 (#5036)

FlamingoPg
authored and
thyecust
committed
Revert "Replace enable_flashinfer_mla argument with attention_backend" (#5048)

merrymercy
authored and
thyecust
committed
bump sgl-kernel v0.0.7 (#5046)

zhyncs
authored and
thyecust
committed
update eagle-3 docs (#4796)

authored and
thyecust
committed
Add LlavaLlamaForCausaLM in MultiModal Processors (#5039)

authored and
thyecust
committed
Update the retry count (#5051)

zhyncs
authored and
thyecust
committed
upgrade sgl-kernel v0.0.7 (#5049)

zhyncs
authored and
thyecust
committed
[2/3] fix dsv3 awq issue (#4625)

authored and
thyecust
committed
Feature/revise docs ci (#5056)

renxinx
authored and
thyecust
committed
Add H20 fused MoE kernel tuning configs for DeepSeek V3/R1 (#5057)

M0gician
authored and
thyecust
committed
[fix] remove `cuda_device_count_stateless` (#5060)

Alcanderian
authored and
thyecust
committed
Small refactor DeepEPDispatcher into subclasses (#4994)

fzyzcjy
authored and
thyecust
committed
Support async DeepEP by splitting into two stages (#4995)

fzyzcjy
authored and
thyecust
committed
Cleanup unused resources after DeepEP operation (#4996)

fzyzcjy
authored and
thyecust
committed
Add DeepSeek V3/R1 shared experts fusion (#4918)

BBuf
authored and
thyecust
committed
[deepep] fix: shared experts are not initialized when shared experts fusion is enabled (#5072)

ch-wan
authored and
thyecust
committed
fix dummy-load deepseekv2 (#4535)

inkcherry
authored and
thyecust
committed
support sgl-kernel on blackwell (#5074)

zhyncs
authored and
thyecust
committed
FA3 Spec Decoding to support top k = 1 and add cuda graph support (#5050)

authored and
thyecust
committed
[Revision] Replace enable_flashinfer_mla argument with attention_backend (#5052)

Fridge003
authored and
thyecust
committed
upgrade transformers 4.51.0 (#5088)

zhyncs
authored and
thyecust
committed
sgl-kernel transfer custom allreduce from trt kernel to vllm kernel (#5079)

yizhang2077
authored and
thyecust
committed
bump sgl-kernel 0.0.8 (#5089)

zhyncs
authored and
thyecust
committed
python transfer custom allreduce from trt kernel to vllm kernel (#5080)

yizhang2077
authored and
thyecust
committed
bump v0.4.4.post4 (#5091)

zhyncs
authored and
thyecust
committed
Fix: Reduce the number of document ci attempts to avoid long ci running (#5097)

authored and
thyecust
committed
Add Llama4 support (#5092)

authored and
thyecust
committed
Fix refactor error - fp8.py (#5106)

authored and
thyecust
committed
bump v0.4.5 (#5117)

zhyncs
authored and
thyecust
committed
[ci] fix llama4 ci error (#5126)

BBuf
authored and
thyecust
committed
Refactor and Optimize FA3 Code (#5090)

authored and
thyecust
committed
Add Llama4 user guide (#5133)

authored and
thyecust
committed
[Misc] Use pytest.mark.skipif in sgl-kernel test (#5137)

FlamingoPg
authored and
thyecust
committed
feat: disable grammar restrictions within reasoning sections (#4984)

committed
[modelopt] automatically inspect if model is ModelOpt quantized and set quantization method (#5145)

yundai424
authored and
thyecust
committed
[AMD] Fix missing per_token_group_quant_fp8 for ROCm (#5140)

hubertlu-tw
authored and
thyecust
committed
fix multimodal hash feature (#5083)

huangtingwei9988
authored and
thyecust
committed
Fix run time error in ROCm platform (#5147)

authored and
thyecust
committed
[FA3 Feature] Support multi modal Llama-3.2-11B-Vision-Instruct (#5103)

zcnrex
authored and
thyecust
committed
Add unit test on page_size > 1 and mla and integration test for Flash Attention 3 (#4760)

yubofredwang
authored and
thyecust
committed
Use public model for FA3 speculative decode testing (#5152)

yubofredwang
authored and
thyecust
committed
Add dummy grok test to amd CI. (#5115)

saienduri
authored and
thyecust
committed
fix empty_cache error in pt_weights_iterator (#5151)

authored and
thyecust
committed
Fix torch compile errors (#5158)

kkHuang-amd
authored and
thyecust
committed
Fix loading KV quantization scale; Enable modelopt kv cache (#4686)

authored and
thyecust
committed
[PD] Fix unclosed prefill connection warning of mini_lb (#5155)

ShangmingCai
authored and
thyecust
committed
Add optimized native kernels in sgl-kernel (#5150)

authored and
thyecust
committed
[PD] Simplify mini LB (#4911)

authored and
thyecust
committed
Small improvement of native api docs (#5139)

authored and
thyecust
committed
[feat&refactor] Enhance multimodal input support with refactor io_struct (#4938)

JustinTong0323
authored and
thyecust
committed
Support 2x8xH100 for Llama 4 (#5159)

fzyzcjy
authored and
thyecust
committed
FP4 weight loading and inference (2/2) (#3972)

trevor-m
authored and
thyecust
committed
Fix multimodal hashing error (#5174)

fzyzcjy
authored and
thyecust
committed
Tiny disable model that does not work (#5175)

fzyzcjy
authored and
thyecust
committed
[Bugfix] Fix index out of bounds in local attention with large sequences (#5173)

CatherineSue
authored and
thyecust
committed
[Fix] DeepEP Compatibility with Low Latency (#5068)

authored and
thyecust
committed
docs: remove the use of Downward API for LWS_WORKER_INDEX (#5110)

yankay
authored and
thyecust
committed
feat: add DeepGEMM build warning (#5176)

authored and
thyecust
committed
fix: use DeepEPDispatcher on CUDA (#5180)

zhyncs
authored and
thyecust
committed
[DeepEP] fix: import buffer error (#5179)

ch-wan
authored and
thyecust
committed
Let `bench_one_batch` support `enable_dp_attention` (#4058)

fzyzcjy
authored and
thyecust
committed
[Misc] clean up vllm in sgl-kernel test (#5189)

FlamingoPg
authored and
thyecust
committed
Fix ci test "test_eval_fp8_accuracy" failed (#5185)

authored and
thyecust
committed
Optimize topk operation in llama4 (#5128)

fzyzcjy
authored and
thyecust
committed
Support Llama4 fp8 inference (#5194)

authored and
thyecust
committed
[ci] fix ci test fused_moe op (#5102)

BBuf
authored and
thyecust
committed
model: support mllama4 (#5144)

mickqian
authored and
thyecust
committed
update grok test (#5171)

saienduri
authored and
thyecust
committed
sgl-kernel use cutlass latest version for fp8 blockwise gemm (#5207)

yizhang2077
authored and
thyecust
committed
Add H20 dtype fp8_w8a8 fused MoE kernel tuning configs for DeepSeek V3/R1 (#5196)

Muuuchen
authored and
thyecust
committed
fix: log warning when disable cuda graph (#5209)

zhyncs
authored and
thyecust
committed
[metrics] Add in queue metrics (#4444)

hebiao064
authored and
thyecust
committed
Fix DeepSeek error when using DeepEP mode (#5190)

fzyzcjy
authored and
thyecust
committed
reduce moe_align_block_size_kernel small batch mode overhead (#5086)

BBuf
authored and
thyecust
committed
[PD] Support KV transfer with mooncake (#4880)

authored and
thyecust
committed
[PD] Add get_contiguous_buf_infos interface for MLATokenToKVPool (#5204)

stmatengss
authored and
thyecust
committed
Update deps for mllama4 (#5215)

ispobock
authored and
thyecust
committed
Fix deepseek-v3 with torch.compile in PyTorch 2.6. (#5213)

zou3519
authored and
thyecust
committed
ROCm sgl-kernel: compatible to later torch (#5167)

HaiShaw
authored and
thyecust
committed
[Misc] Clean sgl-kernel test (#5216)

FlamingoPg
authored and
thyecust
committed
Update Makefile / build script to avoid installing incompatible torch dependency (#5245)

elfiegg
authored and
thyecust
committed
Fix torch.compile cacheing (#5259)

authored and
thyecust
committed
ROCm/AITER CK_MoE: update 2-stage kernels & support both Activations (#5228)

HaiShaw
authored and
thyecust
committed
Optimize attention in llama4 (#5127)

fzyzcjy
authored and
thyecust
committed
Optimize GPU memory usage in FlashAttentionBackend's strided indexing (#5262)

authored and
thyecust
committed
Support `--enable-llama4-multimodal` (#5254)

ch-wan
authored and
thyecust
committed
[fix] fix mrope positions not picked up (#5265)

mickqian
authored and
thyecust
committed
doc: nested loop code for offline engine (#5244)

minleminzui
authored and
thyecust
committed
fix: examples for token_in_token_out_vlm (#5193)

JustinTong0323
authored and
thyecust
committed
Fix a 404 link in send_request.ipynb (#5280)

windsonsea
authored and
thyecust
committed
fix: enable fp4 compilation on cu128 (#5286)

zhyncs
authored and
thyecust
committed
update
thyecust
committed

Comments

Commits

Commits on Apr 11, 2025