Skip to content

[Bug] runner freezing for kimi-k2.6 model for some inputs #23676

Open
Open
[Bug] runner freezing for kimi-k2.6 model for some inputs#23676
@xhejtman

Description

@xhejtman
xhejtman
opened on Apr 24, 2026

Checklist

  • I searched related issues but found no solution.
  • The bug persists in the latest version.
  • Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
  • If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
  • Please use English. Otherwise, it will be closed.

Describe the bug

Hello,

today I noticed that running sglang (even nightly build) with kimi-k2.6 causes main proces freeze, probably after receiving some specific input. This is output from py-spy, when it happens:

  %Own   %Total  OwnTime  TotalTime  Function (filename)
 98.00%  98.00%    1.98s     1.98s   _worker (concurrent/futures/thread.py)
 98.00%  98.00%    1.98s     1.98s   monitor (cpu_monitor.py)
 87.00%  91.00%    1.67s     1.77s   added_tokens_encoder (transformers/tokenization_python.py)
  5.00%  96.00%   0.170s     1.95s   _convert_token_to_id_with_added_voc (transformers/tokenization_python.py)
  4.00%   4.00%   0.100s    0.100s   <lambda> (transformers/tokenization_python.py)
  4.00%   4.00%   0.070s    0.070s   poll (multiprocessing/popen_fork.py)
  2.00%  98.00%   0.030s     1.98s   convert_tokens_to_ids (transformers/tokenization_utils_base.py)
  0.00%   0.00%   0.010s    0.010s   _convert_token_to_id (modules/transformers_modules/_2b2b88e33c96d813ebb4c83a740252fe08018e3a/tokenization_kimi.py)
  0.00%  98.00%   0.000s     1.98s   <module> (sglang)
  0.00%  98.00%   0.000s     1.98s   __call__ (starlette/routing.py)
  0.00%  98.00%   0.000s     1.98s   apply_chat_template (modules/transformers_modules/_2b2b88e33c96d813ebb4c83a740252fe08018e3a/tokenization_kimi.py)
  0.00%  98.00%   0.000s     1.98s   handle_request (openai/serving_base.py)
  0.00%  98.00%   0.000s     1.98s   __call__ (uvicorn/middleware/proxy_headers.py)
  0.00%   4.00%   0.000s    0.070s   _monitor_loop (utils/watchdog.py)
  0.00%  98.00%   0.000s     1.98s   run_asgi (uvicorn/protocols/http/h11_impl.py)
  0.00%  98.00%   0.000s     1.98s   app (fastapi/routing.py)
  0.00%  98.00%   0.000s     1.98s   call_next (utils/http_middleware_patch.py)
  0.00%   4.00%   0.000s    0.070s   _check_processes (utils/watchdog.py)
  0.00%  98.00%   0.000s     1.98s   _convert_to_internal_request (openai/serving_chat.py)
  0.00%  98.00%   0.000s     1.98s   _process_messages (openai/serving_chat.py)
  0.00%  98.00%   0.000s     1.98s   run_server (sglang/launch_server.py)
  0.00% 200.00%   0.000s     4.03s   _bootstrap_inner (threading.py)
  0.00%  98.00%   0.000s     1.98s   run (asyncio/runners.py)
  0.00%  98.00%   0.000s     1.98s   get_input_ids (transformers/tokenization_python.py)
  0.00%  98.00%   0.000s     1.98s   handle (starlette/routing.py)
  0.00%  98.00%   0.000s     1.98s   main (sglang/cli/main.py)
  0.00%  98.00%   0.000s     1.98s   _apply_jinja_template (openai/serving_chat.py)
  0.00%  98.00%   0.000s     1.98s   run (uvicorn/main.py)
  0.00%  98.00%   0.000s     1.98s   openai_v1_chat_completions (http_server.py)

this process runs on 100 % cpu, metrics are not working, same for running new requests. Restart helps for some time.

it is the process: /usr/bin/python3 /usr/local/bin/sglang serve

Reproduction

Run kimi-k2.6. Got stuck after some time.

Environment

Python: 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA B200
GPU 0,1,2,3,4,5,6,7 Compute Capability: 10.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 13.0, V13.0.88
CUDA Driver Version: 595.58.03
PyTorch: 2.9.1+cu130
sglang: 0.0.0.dev1+gcbc2bee54
sglang-kernel: 0.4.1+cu130
flashinfer_python: 0.6.8.post1
flashinfer_cubin: 0.6.8.post1
flashinfer_jit_cache: 0.6.8.post1+cu130
triton: 3.5.1
transformers: 5.5.4
torchao: 0.17.0+cu130
numpy: 2.3.5
aiohttp: 3.13.5
fastapi: 0.136.1
huggingface_hub: 1.11.0
interegular: 0.3.3
modelscope: 1.36.1
orjson: 3.11.8
outlines: 0.1.11
packaging: 26.1
psutil: 7.2.2
pydantic: 2.13.3
python-multipart: 0.0.26
pyzmq: 27.1.0
uvicorn: 0.46.0
uvloop: 0.22.1
vllm: Module Not Found
xgrammar: 0.1.32
openai: 2.6.1
tiktoken: 0.12.0
anthropic: 0.97.0
litellm: Module Not Found
torchcodec: 0.9.1+cu130
NVIDIA Topology: 
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    NIC8    NIC9    NIC10   NIC11   NIC12   NIC13   NIC14   NIC15   CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    NODE    NODE    NODE    NODE    PXB     NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    NODE    NODE    NODE    NODE    NODE    NODE    PXB     NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE    PXB     NODE    SYS     SYS     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE    PXB     SYS     SYS     SYS     SYS     SYS     SYS     0-55,112-167    0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PXB     NODE    NODE    NODE    NODE    NODE    56-111,168-223  1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    PXB     NODE    NODE    56-111,168-223  1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    PXB     NODE    56-111,168-223  1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    PXB     56-111,168-223  1               N/A
NIC0    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      PIX     PIX     PIX     NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS
NIC1    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     PIX      X      PIX     PIX     NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS
NIC2    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     PIX     PIX      X      PIX     NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS
NIC3    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     PIX     PIX     PIX      X      NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS
NIC4    PXB     NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE     X      NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS
NIC5    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE     X      PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS
NIC6    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    PIX      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS
NIC7    NODE    PXB     NODE    NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    NODE    NODE     X      NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS
NIC8    NODE    NODE    PXB     NODE    SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE     X      NODE    SYS     SYS     SYS     SYS     SYS     SYS
NIC9    NODE    NODE    NODE    PXB     SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE     X      SYS     SYS     SYS     SYS     SYS     SYS
NIC10   SYS     SYS     SYS     SYS     PXB     NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    NODE    NODE
NIC11   SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE     X      PIX     NODE    NODE    NODE
NIC12   SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    PIX      X      NODE    NODE    NODE
NIC13   SYS     SYS     SYS     SYS     NODE    PXB     NODE    NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      NODE    NODE
NIC14   SYS     SYS     SYS     SYS     NODE    NODE    PXB     NODE    SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE     X      NODE
NIC15   SYS     SYS     SYS     SYS     NODE    NODE    NODE    PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     NODE    NODE    NODE    NODE    NODE     X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9
  NIC10: mlx5_10
  NIC11: mlx5_11
  NIC12: mlx5_12
  NIC13: mlx5_13
  NIC14: mlx5_14
  NIC15: mlx5_15


ulimit soft: 1048576

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions