Add Support for Qwen2-VL Multi-modal Embedding Models

[Titan-p]

Add Support for Qwen2-VL Multi-modal Embedding Models

Motivation

This PR introduces multi-modal embedding capabilities to support the Alibaba-NLP/gme-Qwen2-VL-2B-Instruct model, enabling unified processing of both text and image inputs.

Modifications

1. Model Integration

   • Added model launch configuration for gme-Qwen2-VL
   • Implemented new conversation template

2. API modification

   • v1/embeddings support text/image
   • Usage Example
     • payload = json.dumps({ "input": [ { "text": "text string" }, { "image": "/home/panlyu/images/006.jpg" } ] })

TODO

1. fused input support

Checklist

• Formatted code using pre-commit hooks
• Updated API documentation with image embedding examples
• Verified throughput with mixed text/image batches
• Benchmark results will be shared in Slack channel

[zhaochenyang20]

#3772

@Titan-p What's the difference?

[zhaochenyang20]

@Titan-p Please add test to it. And might @mickqian and @yizhang2077 take a look? Thanks a lot!

[Titan-p]

@Titan-p Please add test to it. And might @mickqian and @yizhang2077 take a look? Thanks a lot!

Unit test added.

[zhaochenyang20]

It should be okay right now. But later we need to refactor this. cc @yizhang2077 Do you agree with this? @mickqian

[zhaochenyang20]

@simveit Could you continue help on this? Thanks so much. If you feel okay. I can merge this

[simveit]

@simveit Could you continue help on this? Thanks so much. If you feel okay. I can merge this

I will go over the code one more time and test it tonight also @zhaochenyang20

[simveit]

The test failed

texts similarity diff tensor(1.9073e-06)
images similarity diff tensor(0.0001)
F
======================================================================
FAIL: test_accuracy (__main__.TestQmeQwenModels)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/misc/simon/sglang/test/srt/models/test_gme_qwen_models.py", line 81, in test_accuracy
    self.assert_close_embeddings(model, prefill_tolerance, torch_dtype)
  File "/home/misc/simon/sglang/test/srt/models/test_gme_qwen_models.py", line 74, in assert_close_embeddings
    assert torch.all(
AssertionError: embeddings are not all close

Before merging it we should increase prefill_tolerance. @zhaochenyang20 @Titan-p

[Titan-p]

The test failed

texts similarity diff tensor(1.9073e-06)
images similarity diff tensor(0.0001)
F
======================================================================
FAIL: test_accuracy (__main__.TestQmeQwenModels)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/misc/simon/sglang/test/srt/models/test_gme_qwen_models.py", line 81, in test_accuracy
    self.assert_close_embeddings(model, prefill_tolerance, torch_dtype)
  File "/home/misc/simon/sglang/test/srt/models/test_gme_qwen_models.py", line 74, in assert_close_embeddings
    assert torch.all(
AssertionError: embeddings are not all close

Before merging it we should increase prefill_tolerance. @zhaochenyang20 @Titan-p

I think it might be related to differences in the image processing. Could you please provide the test images?

[simveit]

The test failed

texts similarity diff tensor(1.9073e-06)
images similarity diff tensor(0.0001)
F
======================================================================
FAIL: test_accuracy (__main__.TestQmeQwenModels)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/misc/simon/sglang/test/srt/models/test_gme_qwen_models.py", line 81, in test_accuracy
    self.assert_close_embeddings(model, prefill_tolerance, torch_dtype)
  File "/home/misc/simon/sglang/test/srt/models/test_gme_qwen_models.py", line 74, in assert_close_embeddings
    assert torch.all(
AssertionError: embeddings are not all close

Before merging it we should increase prefill_tolerance. @zhaochenyang20 @Titan-p

I think it might be related to differences in the image processing. Could you please provide the test images?

i ran it with default on a100

[zhaochenyang20]

also, fix the confilictws

[Titan-p]

cc @zhaochenyang20 @simveit. I think this PR is ready to be merged.

[zhaochenyang20]

I tested it locally and LGTM.

[simveit]

@zhaochenyang20 i will integrate an corresponding example for this over the weekend.

[zhaochenyang20]

@simveit Thansk1