[VLM] Replace torch.repeat_interleave with faster np.repeat for Qwen-VL series

[yuan-luo]

Motivation

Replace torch.repeat_interleave with faster np.repeat for Qwen2-VL, Qwen3-VL, Qwen3-VL-MoE.
E2E TTFT reduces 1.5%.
gsm8k matched. More benchmark and profile will be updated.

[The result is from commit 1, which is updated in commit 2. But it's obvious to show the performance improvement, so keep it here.]
$python test/srt/ops/test_repeat_interleave.py 
[2025-11-21 23:59:33] INFO utils.py:148: Note: detected 192 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
[2025-11-21 23:59:33] INFO utils.py:151: Note: NumExpr detected 192 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
[2025-11-21 23:59:33] INFO utils.py:164: NumExpr defaulting to 16 threads.
CPU correctness check: PASSED.

All correctness checks passed. Starting benchmark...

=== CPU benchmark ===
[CPU] T=  128, iters=2000 | torch=  54.57 ms, numpy=  18.58 ms, same=True
[CPU] T=  512, iters=2000 | torch=  61.48 ms, numpy=  26.38 ms, same=True
[CPU] T= 2048, iters=2000 | torch=  92.67 ms, numpy=  65.20 ms, same=True
[CPU] T= 8192, iters=2000 | torch= 299.10 ms, numpy= 279.42 ms, same=True

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process

[gemini-code-assist]

Summary of Changes

Hello @yuan-luo, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on enhancing the performance of Qwen-VL series models by optimizing a specific tensor operation. It transitions from a PyTorch-native repeat_interleave function to a more efficient NumPy equivalent for calculating cu_seqlens, resulting in measurable speed improvements for these vision-language models.

Highlights

• Performance Optimization: Replaced torch.repeat_interleave with np.repeat for cu_seqlens computation in Qwen-VL series models, leading to a 1.5% reduction in End-to-End Time To First Token (TTFT).
• New Utility Function: Introduced a new utility function compute_cu_seqlens_from_grid_numpy in sglang/srt/models/utils.py to encapsulate the NumPy-based cu_seqlens calculation.
• Benchmarking and Testing: Added a dedicated test file test/srt/ops/test_repeat_interleave.py to verify correctness and benchmark the performance gains of np.repeat over torch.repeat_interleave on CPU.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request replaces torch.repeat_interleave with the faster np.repeat for computing cu_seqlens in Qwen-VL models, backed by benchmark results. The changes include refactoring the logic into a utility function and adding a comprehensive test file. My review focuses on a potential integer overflow issue in the new numpy-based implementation and suggests a fix to ensure correctness for large inputs. I also noted an opportunity to reduce code duplication by using the new utility function in all relevant models.

[yhyang201]

LGTM. I just wanted to check — do we need to add it to the unit tests?

[yuan-luo]

LGTM. I just wanted to check — do we need to add it to the unit tests?

This function can be reused by more models. Adding it to unit test seems reasonable. We can do more enhancement in terms of model ops. This unit test can be an example.

[BBuf]

Please refactor the test into a unit-test style, keeping it consistent with the other tests under the srt directory.

[yuan-luo]

Refactored test to unit-test style.

$python -m pytest ./test/srt/ops/test_repeat_interleave.py
/opt/conda/lib/python3.10/site-packages/pytest_asyncio/plugin.py:252: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

  warnings.warn(PytestDeprecationWarning(_DEFAULT_FIXTURE_LOOP_SCOPE_UNSET))
============================================================================================= test session starts =============================================================================================
platform linux -- Python 3.10.13, pytest-8.3.5, pluggy-1.5.0
rootdir: /root/luoyuan.luo/workspace/sglang_dev
plugins: anyio-4.11.0, asyncio-1.2.0
asyncio: mode=strict, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 41 items                                                                                                                                                                                            

test_repeat_interleave.py .........................................                                                                                                                                     [100%]

============================================================================================== warnings summary ===============================================================================================
<frozen importlib._bootstrap>:241
  <frozen importlib._bootstrap>:241: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:241
  <frozen importlib._bootstrap>:241: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================================================= 41 passed, 2 warnings in 6.46s ========================================================================================
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute