[Feature] Cutlass kernels for LoRA

[lifuhuang]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Creating an issue to track the work for supporting a CUTLASS / CUTE kernel for LoRA to see if there is any perf gain comparing with the current Triton one.

Dependency: this task should happen after #7809 as the FlashInfer deprecation is expected to change / simplify the kernel interface.

(cc @Fridge003 @Ying1123 )

Related resources

No response

[Fridge003]

We might adapt grouped gemm kernel from Cutlass examples:
https://github.com/NVIDIA/cutlass/blob/main/examples/24_gemm_grouped/gemm_grouped.cu

The gemv kernel from Punica might also be useful.
https://github.com/punica-ai/punica/blob/master/csrc/sgmv/sgmv_cutlass.cuh

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.