[Feature] Support for Hunyuan-A13B-Instruct

[RodriMora]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Tencent released this new model:
https://huggingface.co/tencent/Hunyuan-A13B-Instruct

It matches bigger models on benchmarks. It has a decent size to run locally and the MoE architecture should make it pretty fast.
It has 256K context too.

The tencent team released a docker version compatible with sglang, but it would be nice if native support was added.

Related resources

No response

[yilian49]

I believe hunyuan support has been implemented in #7549.

[RodriMora]

I believe hunyuan support has been implemented in #7549.

sorry I tested with the latest release (v0.4.8) but that commit wasn't merged yet.

Anyways I tested with that PR but at least the GPQT model doesn't work:

Step:

gh pr checkout 7549
python3.12 -m venv .venv
source .venv/bin/activate.fish
python -m pip install --upgrade pip
python -m pip install uv
python -m uv pip install -e "python[all]"
python -m sglang.launch_server --model-path tencent/Hunyuan-A13B-Instruct-GPTQ-Int4 --tp 4 --context-length 2048 --port 5000 --host 0.0.0.0 --trust-remote-code

Error output:

Details

[2025-06-27 18:50:45] server_args=ServerArgs(model_path='/mnt/llms/models/tencent/Hunyuan-A13B-Instruct-GPTQ-Int4/', tokenizer_path='/mnt/llms/models/tencent/Hunyuan-A13B-Instruct-GPTQ-Int4/', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=2048, device='cuda', served_model_name='/mnt/llms/models/tencent/Hunyuan-A13B-Instruct-GPTQ-Int4/', chat_template=None, completion_template=None, is_embedding=False, enable_multimodal=None, revision=None, impl='auto', host='0.0.0.0', port=5001, mem_fraction_static=0.867, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=2048, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=4, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=583376357, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, bucket_time_to_first_token=None, bucket_e2e_request_latency=None, bucket_inter_token_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_d
raft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_moe=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, cuda_graph_max_bs=80, cuda_graph_bs=None, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_nccl_nvls=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=False, disable_overlap_cg_plan=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, warmups=None, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntuai/sglang/sglang/python/sglang/launch_server.py", line 14, in <module>
    launch_server(server_args)
  File "/home/ubuntuai/sglang/sglang/python/sglang/srt/entrypoints/http_server.py", line 834, in launch_server
    tokenizer_manager, template_manager, scheduler_info = _launch_subprocesses(
                                                          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/sglang/python/sglang/srt/entrypoints/engine.py", line 756, in _launch_subprocesses
    tokenizer_manager = TokenizerManager(server_args, port_args)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/sglang/python/sglang/srt/managers/tokenizer_manager.py", line 196, in __init__
    self.model_config = ModelConfig.from_server_args(server_args)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/sglang/python/sglang/srt/configs/model_config.py", line 257, in from_server_args
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/sglang/python/sglang/srt/configs/model_config.py", line 243, in __init__
    self._verify_quantization()
  File "/home/ubuntuai/sglang/sglang/python/sglang/srt/configs/model_config.py", line 395, in _verify_quantization
    quantization_override = method.override_quantization_method(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/sglang/python/sglang/srt/layers/quantization/gptq.py", line 284, in override_quantization_method
    can_convert = cls.is_gptq_marlin_compatible(hf_quant_cfg)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/sglang/python/sglang/srt/layers/quantization/gptq.py", line 347, in is_gptq_marlin_compatible
    return check_marlin_supported(
           ^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'check_marlin_supported' is not defined. Did you mean: 'check_marlin_format'?

[KMouratidis]

Seems to be the same error as #7583.

[lcy4869]

It looks the function is imported from vllm. May need to have vllm

[zhyncs]

You should use pip install sglang==0.4.8.post1.

[RodriMora]

You should use pip install sglang==0.4.8.post1.

I still have the same error with that version:
NameError: name 'check_marlin_supported' is not defined. Did you mean: 'check_marlin_format'?

python3.12 -m venv .venv
source .venv/bin/activate.fish
python -m pip install --upgrade pip
python -m pip install uv
python -m uv pip install "sglang[all]>=0.4.8.post1"
CUDA_VISIBLE_DEVICES=0,1,3,5 python -m sglang.launch_server --model-path tencent/Hunyuan-A13B-Instruct-GPTQ-Int4 --tp 4 --context-length 2048 --port 5000 --host 0.0.0.0 --trust-remote-code

python3 -m sglang.check_env

Details

/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/torch/cuda/__init__.py:287: UserWarning:
NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.
If you want to use the NVIDIA GeForce RTX 5090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(
Python: 3.12.11 (main, Jun  4 2025, 08:56:18) [GCC 11.4.0]
CUDA available: True
GPU 0,1,3,5: NVIDIA GeForce RTX 3090
GPU 2,4: NVIDIA GeForce RTX 5090
GPU 0,1,3,5 Compute Capability: 8.6
GPU 2,4 Compute Capability: 12.0
CUDA_HOME: /usr/local/cuda-12.8
NVCC: Cuda compilation tools, release 12.8, V12.8.93
CUDA Driver Version: 575.57.08
PyTorch: 2.7.1+cu126
sglang: 0.4.8.post1
sgl_kernel: 0.1.9
flashinfer_python: 0.2.6.post1
triton: 3.3.1
transformers: 4.52.3
torchao: 0.9.0
numpy: 2.3.1
aiohttp: 3.12.13
fastapi: 0.115.14
hf_transfer: 0.1.9
huggingface_hub: 0.33.1
interegular: 0.3.3
modelscope: 1.27.1
orjson: 3.10.18
outlines: 0.1.11
packaging: 25.0
psutil: 7.0.0
pydantic: 2.11.7
python-multipart: 0.0.20
pyzmq: 27.0.0
uvicorn: 0.34.3
uvloop: 0.21.0
vllm: Module Not Found
xgrammar: 0.1.19
openai: 1.93.0
tiktoken: 0.9.0
anthropic: 0.55.0
litellm: 1.73.2
decord: 0.6.0
NVIDIA Topology:
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    NODE    NODE    NODE    NODE    0-47    0               N/A
GPU1    NODE     X      NODE    NODE    NODE    NODE    0-47    0               N/A
GPU2    NODE    NODE     X      PHB     NODE    NODE    0-47    0               N/A
GPU3    NODE    NODE    PHB      X      NODE    NODE    0-47    0               N/A
GPU4    NODE    NODE    NODE    NODE     X      PHB     0-47    0               N/A
GPU5    NODE    NODE    NODE    NODE    PHB      X      0-47    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

ulimit soft: 1024

Full error output log:

Details

CUDA_VISIBLE_DEVICES=0,1,3,5 python -m sglang.launch_server --model-path /mnt/llms/models/tencent/Hunyuan-A13B-Instruct-GPTQ-Int4/ --tp 4 --context-length 2048 --port 5001 --host 0.0.0.0 --trust-remote-code
[2025-06-28 15:29:38] server_args=ServerArgs(model_path='/mnt/llms/models/tencent/Hunyuan-A13B-Instruct-GPTQ-Int4/', tokenizer_path='/mnt/llms/models/tencent/Hunyuan-A13B-Instruct-GPTQ-Int4/', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=2048, device='cuda', served_model_name='/mnt/llms/models/tencent/Hunyuan-A13B-Instruct-GPTQ-Int4/', chat_template=None, completion_template=None, is_embedding=False, enable_multimodal=None, revision=None, impl='auto', host='0.0.0.0', port=5001, mem_fraction_static=0.867, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=2048, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=4, pp_size=1, max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=124519794, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=0, show_time_cost=False, enable_metrics=False, bucket_time_to_first_token=None, bucket_e2e_request_latency=None, bucket_inter_token_latency=None, collect_tokens_histogram=False, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_d
raft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, ep_size=1, enable_ep_moe=False, enable_deepep_moe=False, enable_flashinfer_moe=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, cuda_graph_max_bs=80, cuda_graph_bs=None, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_nccl_nvls=False, enable_tokenizer_batch_encode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, disable_overlap_schedule=False, disable_overlap_cg_plan=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_torch_compile=False, torch_compile_max_bs=32, torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through_selective', flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, enable_return_hidden_states=False, warmups=None, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, debug_tensor_dump_prefill_only=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, num_reserved_decode_tokens=512, pdlb_url=None, custom_weight_loader=[], weight_loader_disable_mmap=False)
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/launch_server.py", line 14, in <module>
    launch_server(server_args)
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/entrypoints/http_server.py", line 834, in launch_server
    tokenizer_manager, template_manager, scheduler_info = _launch_subprocesses(
                                                          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/entrypoints/engine.py", line 756, in _launch_subprocesses
    tokenizer_manager = TokenizerManager(server_args, port_args)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/managers/tokenizer_manager.py", line 196, in __init__
    self.model_config = ModelConfig.from_server_args(server_args)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/configs/model_config.py", line 257, in from_server_args
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/configs/model_config.py", line 243, in __init__
    self._verify_quantization()
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/configs/model_config.py", line 395, in _verify_quantization
    quantization_override = method.override_quantization_method(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/gptq.py", line 284, in override_quantization_method
    can_convert = cls.is_gptq_marlin_compatible(hf_quant_cfg)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntuai/sglang/.venv/lib/python3.12/site-packages/sglang/srt/layers/quantization/gptq.py", line 347, in is_gptq_marlin_compatible
    return check_marlin_supported(
           ^^^^^^^^^^^^^^^^^^^^^^
NameError: name 'check_marlin_supported' is not defined. Did you mean: 'check_marlin_format'?

Edit: fixed now with #7583 (comment)