Support InternVL3

[xiaomin-D]

Motivation

Support InternVL3

Modifications

Based on PR #3351. and #4433

Support both InternLM2ForCausalLM & Qwen2ForCausalLM as language model.
Support InternVL2.5 and InternVL3

Checklist

• [✅] Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[zhaochenyang20]

@yizhang2077 @mickqian could you help to review on this?

[GeLee-Q]

This modification also applies to qwen2.5-vl, and perhaps it could be adjusted to make it compatible with more models.

[minleminzui]

I tried your test, but it didn’t work. Do you have any advice to help me fix it? thanks @xiaomin-D

root@71b970322e07:/sgl-workspace/sglang_1/test/srt# python3 test_vision_openai_server.py TestInternVL2_5Server
command=python3 -m sglang.launch_server --model-path OpenGVLab/InternVL2_5-2B --trust-remote-code --chat-template internvl-2-5 --host 127.0.0.1 --port 8000
INFO 04-20 14:10:02 [__init__.py:239] Automatically detected platform cuda.
INFO 04-20 14:10:02 [__init__.py:239] Automatically detected platform cuda.
INFO 04-20 14:10:02 [__init__.py:239] Automatically detected platform cuda.
INFO 04-20 14:10:02 [__init__.py:239] Automatically detected platform cuda.
INFO 04-20 14:10:03 [__init__.py:239] Automatically detected platform cuda.
[2025-04-20 14:10:04] server_args=ServerArgs(model_path='OpenGVLab/InternVL2_5-2B', tokenizer_path='OpenGVLab/InternVL2_5-2B', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=True, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='OpenGVLab/InternVL2_5-2B', chat_template='internvl-2-5', completion_template=None, is_embedding=False, revision=None, host='127.0.0.1', port=8000, mem_fraction_static=0.88, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, stream_interval=1, stream_output=False, random_seed=723275796, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_llama4_multimodal=None, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_ep_moe=False, enable_deepep_moe=False, deepep_mode='auto', enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, hicache_ratio=2.0, flashinfer_mla_disable_ragged=False, warmups=None, n_share_experts_fusion=0, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_bootstrap_port=8998, disaggregation_transfer_backend='mooncake')
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.05k/4.05k [00:00<00:00, 25.2MB/s]
configuration_internvl_chat.py: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.09k/4.09k [00:00<00:00, 38.0MB/s]
configuration_internlm2.py: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.00k/7.00k [00:00<00:00, 19.1MB/s]
A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/InternVL2_5-2B:
- configuration_internlm2.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
configuration_intern_vit.py: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.55k/5.55k [00:00<00:00, 15.1MB/s]
A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/InternVL2_5-2B:
- configuration_intern_vit.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/InternVL2_5-2B:
- configuration_internvl_chat.py
- configuration_internlm2.py
- configuration_intern_vit.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
[2025-04-20 14:10:04] vision_select_layer: -1
[2025-04-20 14:10:04] ps_version: v2
[2025-04-20 14:10:04] min_dynamic_patch: 1
[2025-04-20 14:10:04] max_dynamic_patch: 12
[2025-04-20 14:10:04] Ignore import error when loading sglang.srt.managers.multimodal_processors.gemma3: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
cannot import name 'Int4WeightOnlyConfig' from 'torchao.quantization' (/usr/local/lib/python3.10/dist-packages/torchao/quantization/__init__.py)
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.internvl: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
cannot import name 'Int4WeightOnlyConfig' from 'torchao.quantization' (/usr/local/lib/python3.10/dist-packages/torchao/quantization/__init__.py)
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.janus_pro: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
cannot import name 'Int4WeightOnlyConfig' from 'torchao.quantization' (/usr/local/lib/python3.10/dist-packages/torchao/quantization/__init__.py)
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.llava: Failed to import transformers.models.clip.modeling_clip because of the following error (look up to see its traceback):
/usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.minicpm: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
/usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.mlama: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
cannot import name 'Int4WeightOnlyConfig' from 'torchao.quantization' (/usr/local/lib/python3.10/dist-packages/torchao/quantization/__init__.py)
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.mllama4: Failed to import transformers.models.llama4.modeling_llama4 because of the following error (look up to see its traceback):
/usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.qwen_vl: /usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
preprocessor_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 287/287 [00:00<00:00, 1.01MB/s]
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.01k/4.01k [00:00<00:00, 14.0MB/s]
tokenization_internlm2.py: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8.79k/8.79k [00:00<00:00, 24.1MB/s]
A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/InternVL2_5-2B:
- tokenization_internlm2.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
tokenizer.model: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.48M/1.48M [00:00<00:00, 18.0MB/s]
added_tokens.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 179/179 [00:00<00:00, 668kB/s]
special_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 844/844 [00:00<00:00, 3.22MB/s]
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/sgl-workspace/sglang_1/python/sglang/launch_server.py", line 14, in <module>
    launch_server(server_args)
  File "/sgl-workspace/sglang_1/python/sglang/srt/entrypoints/http_server.py", line 704, in launch_server
    tokenizer_manager, scheduler_info = _launch_subprocesses(server_args=server_args)
  File "/sgl-workspace/sglang_1/python/sglang/srt/entrypoints/engine.py", line 569, in _launch_subprocesses
    tokenizer_manager = TokenizerManager(server_args, port_args)
  File "/sgl-workspace/sglang_1/python/sglang/srt/managers/tokenizer_manager.py", line 197, in __init__
    self.mm_processor = get_mm_processor(
  File "/sgl-workspace/sglang_1/python/sglang/srt/managers/multimodal_processor.py", line 61, in get_mm_processor
    raise ValueError(
ValueError: No processor registered for architecture: ['InternVLChatModel'].
Registered architectures: ['CLIPModel', 'DeepseekVL2ForCausalLM']
/usr/lib/python3.10/multiprocessing/resource_tracker.py:104: UserWarning: resource_tracker: process died unexpectedly, relaunching.  Some resources might leak.
  warnings.warn('resource_tracker: process died unexpectedly, '
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
KeyError: '/mp-vo_juru8'
E
======================================================================
ERROR: setUpClass (__main__.TestInternVL2_5Server)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sgl-workspace/sglang_1/test/srt/test_vision_openai_server.py", line 615, in setUpClass
    cls.process = popen_launch_server(
  File "/sgl-workspace/sglang_1/python/sglang/test/test_utils.py", line 453, in popen_launch_server
    raise Exception(f"Server unexpectedly exits ({return_code=}).")
Exception: Server unexpectedly exits (return_code=1).

----------------------------------------------------------------------
Ran 0 tests in 20.022s

FAILED (errors=1)
root@71b970322e07:/sgl-workspace/sglang_1/test/srt# git branch
  client
  doc
  doc_timeout
  extrabody
* internVL
  main
  nest_asyncio
  separate_reasoning
  structural_tag
  unbalance
  verl

[zhaochenyang20]

@xiaomin-D please rebase with main, we gonna merge it pretty quick

[yizhang2077]

some comments

[xiaomin-D]

I tried your test, but it didn’t work. Do you have any advice to help me fix it? thanks @xiaomin-D

root@71b970322e07:/sgl-workspace/sglang_1/test/srt# python3 test_vision_openai_server.py TestInternVL2_5Server
command=python3 -m sglang.launch_server --model-path OpenGVLab/InternVL2_5-2B --trust-remote-code --chat-template internvl-2-5 --host 127.0.0.1 --port 8000
INFO 04-20 14:10:02 [__init__.py:239] Automatically detected platform cuda.
INFO 04-20 14:10:02 [__init__.py:239] Automatically detected platform cuda.
INFO 04-20 14:10:02 [__init__.py:239] Automatically detected platform cuda.
INFO 04-20 14:10:02 [__init__.py:239] Automatically detected platform cuda.
INFO 04-20 14:10:03 [__init__.py:239] Automatically detected platform cuda.
[2025-04-20 14:10:04] server_args=ServerArgs(model_path='OpenGVLab/InternVL2_5-2B', tokenizer_path='OpenGVLab/InternVL2_5-2B', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=True, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='OpenGVLab/InternVL2_5-2B', chat_template='internvl-2-5', completion_template=None, is_embedding=False, revision=None, host='127.0.0.1', port=8000, mem_fraction_static=0.88, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=1, stream_interval=1, stream_output=False, random_seed=723275796, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, decode_log_interval=40, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_llama4_multimodal=None, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_ep_moe=False, enable_deepep_moe=False, deepep_mode='auto', enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, hicache_ratio=2.0, flashinfer_mla_disable_ragged=False, warmups=None, n_share_experts_fusion=0, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_bootstrap_port=8998, disaggregation_transfer_backend='mooncake')
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.05k/4.05k [00:00<00:00, 25.2MB/s]
configuration_internvl_chat.py: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.09k/4.09k [00:00<00:00, 38.0MB/s]
configuration_internlm2.py: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.00k/7.00k [00:00<00:00, 19.1MB/s]
A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/InternVL2_5-2B:
- configuration_internlm2.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
configuration_intern_vit.py: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.55k/5.55k [00:00<00:00, 15.1MB/s]
A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/InternVL2_5-2B:
- configuration_intern_vit.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/InternVL2_5-2B:
- configuration_internvl_chat.py
- configuration_internlm2.py
- configuration_intern_vit.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
[2025-04-20 14:10:04] vision_select_layer: -1
[2025-04-20 14:10:04] ps_version: v2
[2025-04-20 14:10:04] min_dynamic_patch: 1
[2025-04-20 14:10:04] max_dynamic_patch: 12
[2025-04-20 14:10:04] Ignore import error when loading sglang.srt.managers.multimodal_processors.gemma3: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
cannot import name 'Int4WeightOnlyConfig' from 'torchao.quantization' (/usr/local/lib/python3.10/dist-packages/torchao/quantization/__init__.py)
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.internvl: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
cannot import name 'Int4WeightOnlyConfig' from 'torchao.quantization' (/usr/local/lib/python3.10/dist-packages/torchao/quantization/__init__.py)
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.janus_pro: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
cannot import name 'Int4WeightOnlyConfig' from 'torchao.quantization' (/usr/local/lib/python3.10/dist-packages/torchao/quantization/__init__.py)
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.llava: Failed to import transformers.models.clip.modeling_clip because of the following error (look up to see its traceback):
/usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.minicpm: Failed to import transformers.models.llama.modeling_llama because of the following error (look up to see its traceback):
/usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.mlama: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
cannot import name 'Int4WeightOnlyConfig' from 'torchao.quantization' (/usr/local/lib/python3.10/dist-packages/torchao/quantization/__init__.py)
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.mllama4: Failed to import transformers.models.llama4.modeling_llama4 because of the following error (look up to see its traceback):
/usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
[2025-04-20 14:10:05] Ignore import error when loading sglang.srt.managers.multimodal_processors.qwen_vl: /usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC2ENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
preprocessor_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 287/287 [00:00<00:00, 1.01MB/s]
tokenizer_config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.01k/4.01k [00:00<00:00, 14.0MB/s]
tokenization_internlm2.py: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8.79k/8.79k [00:00<00:00, 24.1MB/s]
A new version of the following files was downloaded from https://huggingface.co/OpenGVLab/InternVL2_5-2B:
- tokenization_internlm2.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
tokenizer.model: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.48M/1.48M [00:00<00:00, 18.0MB/s]
added_tokens.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 179/179 [00:00<00:00, 668kB/s]
special_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 844/844 [00:00<00:00, 3.22MB/s]
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/sgl-workspace/sglang_1/python/sglang/launch_server.py", line 14, in <module>
    launch_server(server_args)
  File "/sgl-workspace/sglang_1/python/sglang/srt/entrypoints/http_server.py", line 704, in launch_server
    tokenizer_manager, scheduler_info = _launch_subprocesses(server_args=server_args)
  File "/sgl-workspace/sglang_1/python/sglang/srt/entrypoints/engine.py", line 569, in _launch_subprocesses
    tokenizer_manager = TokenizerManager(server_args, port_args)
  File "/sgl-workspace/sglang_1/python/sglang/srt/managers/tokenizer_manager.py", line 197, in __init__
    self.mm_processor = get_mm_processor(
  File "/sgl-workspace/sglang_1/python/sglang/srt/managers/multimodal_processor.py", line 61, in get_mm_processor
    raise ValueError(
ValueError: No processor registered for architecture: ['InternVLChatModel'].
Registered architectures: ['CLIPModel', 'DeepseekVL2ForCausalLM']
/usr/lib/python3.10/multiprocessing/resource_tracker.py:104: UserWarning: resource_tracker: process died unexpectedly, relaunching.  Some resources might leak.
  warnings.warn('resource_tracker: process died unexpectedly, '
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/resource_tracker.py", line 209, in main
    cache[rtype].remove(name)
KeyError: '/mp-vo_juru8'
E
======================================================================
ERROR: setUpClass (__main__.TestInternVL2_5Server)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/sgl-workspace/sglang_1/test/srt/test_vision_openai_server.py", line 615, in setUpClass
    cls.process = popen_launch_server(
  File "/sgl-workspace/sglang_1/python/sglang/test/test_utils.py", line 453, in popen_launch_server
    raise Exception(f"Server unexpectedly exits ({return_code=}).")
Exception: Server unexpectedly exits (return_code=1).

----------------------------------------------------------------------
Ran 0 tests in 20.022s

FAILED (errors=1)
root@71b970322e07:/sgl-workspace/sglang_1/test/srt# git branch
  client
  doc
  doc_timeout
  extrabody
* internVL
  main
  nest_asyncio
  separate_reasoning
  structural_tag
  unbalance
  verl

docker run \
    -itd \
    --gpus all \
    --privileged --cap-add=IPC_LOCK \
    --ulimit memlock=-1 --ulimit stack=67108864 \
    -v /data0:/data \
    -v /cfs:/cfs \
    --net=host \
    --ipc=host \
    --name=sgldev lmsysorg/sglang:dev

docker exec -it sgldev bash
cd /sgl-workspace && mv sglang sglang_back

cd /sgl-workspace && git clone --branch internVL https://github.com/xiaomin-D/sglang.git
cd /sgl-workspace/sglang/python && pip install -e .

then, launch sglang server and post

python3 -m sglang.launch_server --model-path OpenGVLab/InternVL2_5-2B --trust-remote-code --chat-template internvl-2-5 --host 127.0.0.1 --port 30000


curl -X POST -H "Content-Type: application/json" -d '{
    "model": "",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What’s in this image?"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/tiger.jpeg"
                    }
                }
            ]
        }
    ],
    "max_tokens": 300
}' http://localhost:30000/v1/chat/completions

[yizhang2077]

Some comments

[yizhang2077]

trival comments

[yizhang2077]

LGTM, can we paste a benchmark result here? MMMU or lmms benchmark is available

[yizhang2077]

FYI: Currently InternVL3 inside implement a flashattention for vision model, which do not support TP and may be a bottleneck, we need replace it in the future PR