Skip to content

Comments

[Feature] Support Tensor Parallelism and Weight Slicing for Lora#4239

Closed
aoshen524 wants to merge 264 commits intosgl-project:mainfrom
aoshen524:feature/lora
Closed

[Feature] Support Tensor Parallelism and Weight Slicing for Lora#4239
aoshen524 wants to merge 264 commits intosgl-project:mainfrom
aoshen524:feature/lora

Conversation

@aoshen524
Copy link
Contributor

@aoshen524 aoshen524 commented Mar 9, 2025 •
edited by Fridge003
Loading

Motivation

#3414 reports issues regarding limited model support compared to test_generation_models.py. This PR introduces tensor parallelism and weight slicing for LoRA, alongside additional improvements to testing and functionality.

Modifications

  • Implemented tensor parallelism support in LoRA, allowing efficient distribution of computations across multiple devices.
  • Introduced LoRA weight slicing and refactor memory pool to facilitate distributed inference, optimizing memory usage and performance.

checklist:

  • Remove tensor.contiguous() used in GPU

zhaochenyang20 and others added 13 commits February 19, 2025 16:01
@Fridge003 Fridge003 changed the title Feature/lora [Feature] Support Tensor Parallelism and Weight Slicing for Lora Mar 9, 2025
This was referenced Mar 9, 2025
Closed
Open
merrymercy and others added 22 commits March 11, 2025 04:05
@merrymercy @ShenAo1111
Minor style fix for sgl-kernel (sgl-project#4243)
a43e55a
@merrymercy @ShenAo1111
Auto balance CI tests (sgl-project#4238)
98decee
@merrymercy @ShenAo1111
Clean up fp8 support (sgl-project#4230)
5730180
@merrymercy @ShenAo1111
Move activation.cu to sgl-kernel/elementwise (sgl-project#4250)
110b0ad
@laixinn @sleepcoo
@HandH1998 @shuaills @FlamingoPg @zhyncs @ShenAo1111
DeepGemm integrate to sgl-kernel (sgl-project#4165)
69583c9 
Co-authored-by: sleepcoo <sleepcoo@gmail.com>
Co-authored-by: HandH1998 <1335248067@qq.com>
Co-authored-by: shuaills <shishuaiuoe@gmail.com>
Co-authored-by: yinfan98 <1106310035@qq.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
@DavidChan0519 @ShenAo1111
[Bug fixed] fixed the crash when enable the dp-attention on the singl…
122a4b3 
…e card (sgl-project#3958)
@simveit @zhaochenyang20 @ShenAo1111
Added example for multimodal embedding (sgl-project#4206)
d6666a6 
Co-authored-by: Chayenne <zhaochen20@outlook.com>
@merrymercy @ShenAo1111
Simplify tests & Fix trtllm custom allreduce registration (sgl-projec…
bd58fd8 
…t#4252)
@Young1993 @ShenAo1111
fix the input_ids is None error (sgl-project#4144)
4bf7b08
@BBuf @ShenAo1111
fix per_token_group_quant_fp8 illegal memory when num_groups % 16 != 0 (
5b7a02a 
sgl-project#4231)
@merrymercy @ShenAo1111
Release sgl-kernel v0.0.4.post1 (sgl-project#4255)
9413891
@merrymercy @ShenAo1111
Fix quantization and nightly tests (sgl-project#4258)
95145b6
@merrymercy @ShenAo1111
increase the timeout of nightly-test.yml (sgl-project#4262)
c3b8b97
@merrymercy @ShenAo1111
Optimize rope in sgl kernel (sgl-project#4267)
c2f851e
@merrymercy @ShenAo1111
Test no vllm custom allreduce (sgl-project#4256)
80ba9b3
@HandH1998 @ShenAo1111
Amd test fp8 (sgl-project#4261)
28e811d
@zhyncs @ShenAo1111
add THIRDPARTYNOTICES for DeepGEMM (sgl-project#4272)
2c5b08e
@ShenAo1111
fix(lora): improve handling of gate projections in LoRA weights.
6f1d7d5
@ShenAo1111
style(lora): format LoRA codebase with pre-commit tool.
911b334
@ShenAo1111
perf(sglang): adjust static memory fraction to original value in main…
15ba985 
… branch.
@ShenAo1111
test(srt): update LoRA test tolerances and configurations.
57bc3c7
@ShenAo1111
feat(lora): add support for tensor parallelism in LoRA
dc9ad0a
@Fridge003 Fridge003 closed this Mar 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@merrymercy merrymercy Awaiting requested review from merrymercy merrymercy is a code owner

@Ying1123 Ying1123 Awaiting requested review from Ying1123 Ying1123 is a code owner

@zhyncs zhyncs Awaiting requested review from zhyncs zhyncs is a code owner

@hnyls2002 hnyls2002 Awaiting requested review from hnyls2002 hnyls2002 is a code owner

@ispobock ispobock Awaiting requested review from ispobock ispobock is a code owner

@HaiShaw HaiShaw Awaiting requested review from HaiShaw HaiShaw is a code owner

@HandH1998 HandH1998 Awaiting requested review from HandH1998

@BBuf BBuf Awaiting requested review from BBuf BBuf is a code owner

@yizhang2077 yizhang2077 Awaiting requested review from yizhang2077 yizhang2077 is a code owner

@ByronHsu ByronHsu Awaiting requested review from ByronHsu

@rkooo567 rkooo567 Awaiting requested review from rkooo567

@kssteven418 kssteven418 Awaiting requested review from kssteven418

@zhaochenyang20 zhaochenyang20 Awaiting requested review from zhaochenyang20

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.