[Bug] illegal memory access / illegal instruction / memory leak

[hnyls2002]

Issues

• [Bug] DeepSeek-V3.2: Illegal instruction / IMA during GPQA evaluation with DP attention (H200, TP=8, DP=8) #11942
  • Illegal instruction Fix Illegal Instruction/IMA errors when using DP attention -- num_tokens_for_logprob calculation #12115 (also fix [Bug] token_to_kv_pool_allocator memory leak detected! #11973, [Bug] Memory Leak in token_to_kv_pool_allocator since v0.5.3.post3 (not present in v0.5.3) #11970
  • Illegal memory access Fix Illegal Instruction/IMA errors when using DP attention -- num_tokens_for_logprob calculation #12115
  • Memory leak
• [Bug] Scheduler IMA at high concurrency #11770
  • Illegal memory access
• Remove overlap thread #11210 (comment)
  • Illegal memory access
• [Bug] DeepSeek Infer illegal memory access bugs and dumps #11671

Fix

• [Bug] fix memory leak by overlap + retract #11976
• [Fix] memory leak by overlap + retract #11981

[Swipe4057]

Please enable this feature to track the issue #11860.
After some analysis, I realized that the video memory leak rate also depends on whether you choose xgrammar or llguidance. Also, memory consumption and memory growth are significantly reduced by enabling the disable_custom_all_reduce flag, although the throughput is halved.

[hnyls2002]

@Swipe4057 This is unrelated to the above issues. We can just keep track of it in #11860

[whybeyoung]

can we add #11671

[yudian0504]

Sorry, the previous info ( #11210 (comment) ) might have been misleading. It's actually not just about overlap scheduling - we ran into another IMA issue even after disabling overlap scheduling.

Scheduler hit an exception: Traceback (most recent call last):^M
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2882, in run_scheduler_process^M
    scheduler.event_loop_normal()^M
  File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context^M
    return func(*args, **kwargs)^M
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 954, in event_loop_normal^M
    self.process_batch_result(batch, result)^M
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler.py", line 2121, in process_batch_result^M
    self.process_batch_result_decode(batch, result)^M
  File "/opt/conda/lib/python3.10/site-packages/sglang/srt/managers/scheduler_output_processor_mixin.py", line 238, in process_batch_result_decode^M
    next_token_ids = next_token_ids.tolist()^M
torch.AcceleratorError: CUDA error: an illegal memory access was encountered^M
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.^M
For debugging consider passing CUDA_LAUNCH_BLOCKING=1^M
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.^M

SIGQUIT received. signum=None, frame=None. It usually means one child failed.

[hnyls2002]

@yudian0504 If so, please raise an issue about the error. If this IMA can be reproduced in non-overlap mode, please also try CUDA_LAUNCH_BLOCKING=1 to find the exact kernel.

[sjtushenhai]

Do you have time to take a look this #14363

[hnyls2002]

The previously listed IMA errors are no longer reported recently. Close this summary issue.