[Perf] Profiling and Monitoring VLM performance

[JustinTong0323]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

It is necessary for us to conduct a performance profiling of the VLM through torch profile for the purpose of making a comparison with other frameworks such as vLLM.

Our focus will be placed on the most widely utilized VLMs, namely "OpenGVLab/InternVL3-8B" and "Qwen/Qwen2.5-VL-7B-Instruct".

Related resources

No response

[Swipe4057]

#7471

[AlienKevin]

I'm also interested in VLM performance on SGL. Since most performance difference are expected to be in the prefill stage with current LLaVA-style VLMs, I benchmarked Qwen2.5-VL prefill stage performance on vision-only vs text-only inputs and here are my current takeaways at a high level:

1. Vision processing adds substantial overhead on smaller Qwen variants like 3B and 7B (up to 11.7x on 3B and 6.4x on 7B)

Below is a plot of prefill throughput over a sweep of concurrency settings for vision-only vs text-only inputs on Qwen 7B:

2. Relative overhead from vision is reduced on larger Qwen variants like 32B and 72B as vision encoder size stays the same (up to 2.5x on 32B and 1.7x on 72B)

All benchmarks use a fixed input sequence length of 1000 and output sequence length of 2 to measure prefill stage performance. Text-only experiments use 1000 random text tokens while vision-only experiments use a random image of size 1120x700 that maps to 1000 visual tokens after encoding. Experiments were run on a single B200 GPU with SGL version 0.4.9.post2.

[AlienKevin]

TLDR; Nsys kernel latency breakdown shows that 8.5 out of 32s total GPU time is wasted on host-to-device memcpy in the default VisionSdpaAttention used in Qwen 2.5 VL's vision encoder, with additional 8s spent on slow cutlass AttentionKernel. Switching to FlashAttention3 or Triton attention backends deliver an instant 2x prefill speedup for VisionAttention. SGL should consider automatically setting these efficient attention backends as defaults when no mm-attention-backend option is specified.

Full breakdown:
Below is an nsys profile of one forward pass of SDPA VisionAttention, showing that memcpy and cutlass AttentionKernel each eats up roughly half of the time of vision attention:

Nsys stats reveals that roughly 15MB of data is transferred from host to device during each forward pass of VisionAttention, costing a whopping 910 microseconds:

Looking into the VisionSdpaAttention function, we can pinpoint the origin of the memcpy. Since the experiments are conducted under a batch size of 8 with 4000 input image patches (image size 1120x700), the expected size of the attention mask is 8x4000x4000 bits which is exactly 15.26 MiB! This matches what's logged in Nsys and confirms that even though the masks are lru_cached, they have to be copied to the GPU during each forward pass, which significantly slows down the attention calculation.

sglang/python/sglang/srt/layers/attention/vision.py

Lines 99 to 129 in f8260f2

	def _generate_mask_cache(
	s: int, flatten_batch: bool, cu_seqlens: tuple
	) -> torch.BoolTensor:
	"""
	Generate a boolean attention mask with caching mechanism.
	Args:
	s: sequence length
	flatten_batch: whether to flatten batch dimension
	cu_seqlens: tuple of cumulative sequence lengths
	Returns:
	attention mask tensor of shape [b, 1, s, s] or [1, s, s]
	"""
	if flatten_batch:
	mask = torch.zeros([1, s, s], dtype=torch.bool)
	for i in range(1, len(cu_seqlens)):
	start = cu_seqlens[i - 1]
	end = cu_seqlens[i]
	mask[..., start:end, start:end] = True
	else:
	# [1, 1, 1, s]
	row_indices = torch.arange(s).view(1, 1, 1, s)
	# [1, 1, s, 1]
	col_indices = torch.arange(s).view(1, 1, s, 1)
	# [b, 1, 1, 1]
	seq_lens = torch.tensor(
	[end - start for start, end in zip(cu_seqlens[:-1], cu_seqlens[1:])],
	).view(-1, 1, 1, 1)
	mask = (row_indices < seq_lens) & (col_indices < seq_lens)
	return mask

After switching to the FlashAttention3 or the Triton attention backends, the overall memcpy latency is reduced by >5x and attention calculation is sped up by >9x, resulting in an overall 2x speed up in vision prefill.

Recommendation:
SGL should automatically default to the most performant vision attention backend supported by the machine, if no mm-attention-backend is specified. This is expected to deliver a significant 2x speed up for vision processing.

[JustinTong0323]

@AlienKevin Wow, thank you Kevin!! I'm really happy to hear that we've got this awesome approach that can boost the performance right away! Are you interested to submit a PR to set the default backend and include some profiling comparison? It would be incredibly amazing to see your contribution!

[AlienKevin]

@JustinTong0323 Sure, I'll be happy to prepare a PR!

[AlienKevin]

@JustinTong0323 please see PR: #8484

[Swipe4057]

@AlienKevin
Do you observe any video memory leaks during multimodal workloads in your tests? Perhaps on the B200 you don't notice the growth in video memory consumption and your test ends before the server reboot when OOM occurs.

[AlienKevin]

@Swipe4057 I didn't observe memory related issues on B200 but I did notice CUDA OOM after ~1000 concurrent requests on H200 (ISL=1000, OSL=2), happens for vision attn=SDPA/Triton/FA3. Not sure if this is a memory leak. What do you mean by video memory leak? I tested only on single images.

[Swipe4057]

@AlienKevin Yes, I meant OOM after some time of operation. There is currently a video memory leak when processing multimodal data. If you start tracking process consumption in the console through nvidia-smi, you'll probably notice how one process that appears after the load starts will begin to increase and won't be freed.

[AlienKevin]

SGL is 1.5x slower than vLLM in vision prefill (tested with Qwen 2.5 VL 7B on H200 with Triton vision attention, otherwise 3x slower with SDPA). SGL has less efficient elementwise and GEMM operations and slower memcpy. In my H200 experiments, SGL also runs out of CUDA memory at 1024 concurrency while vLLM cruises to 4096 concurrency with monotonically increasing throughput.

My next steps are to investigate the cause of:

1. 15x higher memcpy latency (1.66s vs 0.11s)
2. 4x slower elementwise (3.48s vs 0.86s)
3. 1.2x slower GEMM (9.30s vs 7.74)
4. CUDA OOM [Bug] VLM cuda memory leak incur OOM #8902

[mickqian]

@AlienKevin Thanks for your detailed and wonderful analysis!

Would mind providing the scripts to reproduce your OOM case? Thx!