Support InternVL2.5

[hehesangsj]

Motivation

Support InternVL2.5

Modifications

Based on PR #3351.

Support both InternLM2ForCausalLM & Qwen2ForCausalLM as language model.

Fix bugs in multi-nodes deployment

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[zhaochenyang20]

@hehesangsj thansk, please fix the conflicts and we can merge this ASAP.

[hehesangsj]

@mickqian @zhaochenyang20 Could you review the code~

[zhaochenyang20]

@mickqian @zhaochenyang20 Could you review the code~

hey, is this #4456 (comment) provided by your team? Otherwise, please provide the benchmark score at #4456

Also, @yizhang2077 could you help to review? thanks so much!

[yizhang2077]

Hi, I leave some simple comments here, I will keep reviewing~ and plz format your code since lint failed.

[yizhang2077]

We provide some attention code in vision.py, could you try to use it instead of writing attention again?

[hehesangsj]

I think it's a little bit hard because the num_heads of InternViT is 25. Using VisionAttention can easily leads to bugs when deploying using --tp 8

[mickqian]

@zhaochenyang20 @yizhang2077 #3351 These are duplicated

[zhaochenyang20]

@zhaochenyang20 @yizhang2077 #3351 These are duplicated

I let them decouple and I will credit you! thanks ! @mickqian

[zhaochenyang20]

@hehesangsj fix the lint and solve the conflicts?

[zhaochenyang20]

@hehesangsj do you sovle all the reviews above? if so, please ping us and we can continue to review. thanks a lot

[hehesangsj]

@hehesangsj do you sovle all the reviews above? if so, please ping us and we can continue to review. thanks a lot

okay, I will solve the reviews ASAP.

[hehesangsj]

@hehesangsj do you sovle all the reviews above? if so, please ping us and we can continue to review. thanks a lot

reviews are fixed and latest main branch is merged. check the code plz~

[zhaochenyang20]

@yizhang2077 yi and @mickqian could you help to review?

[github-actions]

This pull request has been automatically closed due to inactivity. Please feel free to reopen it if needed.