[question] appropriate benchmarks

[keyboardAnt]

Hi, thank you for your awesome contributions to open source!

I'd like to benchmark speculative decoding vocabulary pruning (#3822, https://docs.sglang.ai/backend/speculative_decoding.html#EAGLE-2-Decoding-via-Frequency-Ranked-Speculative-Sampling). What would be the most appropriate benchmarks from https://docs.sglang.ai/references/benchmark_and_profiling.html to use?

In other words, what benchmarks would you like to see before merging a PR that includes an improvement over the existing method?

[keyboardAnt]

#3822 was merged based on results in SpecBench, and I couldn't find any SGLang benchmark results as part of that PR.

1. Is my understanding correct that benchmark offline processing (python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10) is the most similar benchmark to SpecBench?
2. Would the following be a fair benchmark between the existing method (FR-Spec) and future alternatives?

python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \
    --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \
    --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16

For a single A100-SXM4-80GB, I get:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  9.53      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      172.17    
Request throughput (req/s):              1.05      
Input token throughput (tok/s):          205.64    
Output token throughput (tok/s):         291.04    
Total token throughput (tok/s):          496.67    
==================================================

Without speculative decoding (python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10):

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  6.44      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      89.67     
Request throughput (req/s):              1.55      
Input token throughput (tok/s):          304.52    
Output token throughput (tok/s):         430.99    
Total token throughput (tok/s):          735.52    
==================================================

Over the following env (python3 -m sglang.check_env):

Python: 3.10.18 | packaged by conda-forge | (main, Jun  4 2025, 14:45:41) [GCC 13.3.0]
CUDA available: True
GPU 0: NVIDIA A100-SXM4-80GB
GPU 0 Compute Capability: 8.0
CUDA_HOME: /apps/easybd/easybuild/amd/software/CUDA/12.4.0
NVCC: Cuda compilation tools, release 12.4, V12.4.99
CUDA Driver Version: 560.35.05
PyTorch: 2.7.1+cu126
sglang: 0.4.9.post4
sgl_kernel: 0.2.7
flashinfer_python: 0.2.9rc1
triton: 3.3.1
transformers: 4.53.2
torchao: 0.9.0
numpy: 2.2.6
aiohttp: 3.12.14
fastapi: 0.116.1
hf_transfer: 0.1.9
huggingface_hub: 0.34.1
interegular: 0.3.3
modelscope: 1.28.1
orjson: 3.11.1
outlines: 0.1.11
packaging: 25.0
psutil: 7.0.0
pydantic: 2.11.7
python-multipart: 0.0.20
pyzmq: 27.0.0
uvicorn: 0.35.0
uvloop: 0.21.0
vllm: Module Not Found
xgrammar: 0.1.21
openai: 1.97.1
tiktoken: 0.9.0
anthropic: 0.59.0
litellm: 1.74.8
decord: 0.6.0
NVIDIA Topology: 
        GPU0    NIC0    NIC1    NIC2    NIC3    NIC4    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PXB     NODE    NODE    NODE    NODE            1               N/A
NIC0    PXB      X      NODE    NODE    NODE    NODE
NIC1    NODE    NODE     X      SYS     SYS     SYS
NIC2    NODE    NODE    SYS      X      PIX     SYS
NIC3    NODE    NODE    SYS     PIX      X      SYS
NIC4    NODE    NODE    SYS     SYS     SYS      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4


ulimit soft: 66000

[Swipe4057]

Hi, I tested the Fr spec. The testing was conducted as follows: a test dataset was taken that had been previously excluded from the training set of the draft model. Then loads with different sending speeds to the inference server were run sequentially, and graphs were built based on the final points. Some of them are here. The FR spec results are also there. It showed good performance with the Flashinfer+FR graph (reduced overhead on large batch sizes). #6863

[keyboardAnt]

Hi, I tested the Fr spec. The testing was conducted as follows: a test dataset was taken that had been previously excluded from the training set of the draft model. Then loads with different sending speeds to the inference server were run sequentially, and graphs were built based on the final points. Some of them are here. The FR spec results are also there. It showed good performance with the Flashinfer+FR graph (reduced overhead on large batch sizes). #6863

Thank you @Swipe4057!

1. Are there any merged PRs that use your benchmark to demonstrate efficiency gains? In [Bug] FA3 + EAGLE2: speculative_token_map not supported #6863 I only found closed PRs.
2. What does “HASS” stand for?
3. What does the x-y-z notation mean (e.g., “HASS 1-2-2” and “EAGLE2 3-2-4”)?

---

For convenience, I’ve copied your experiment details and results from #6863:

We conducted a large-scale load test with speculative decoding using EAGLE2. According to our results, enabling speculative_token_map effectively reduces overhead for larger batch sizes. However, this feature only works with the flashinfer backend, while the FA3 backend does not support it.

Here is our graph demonstrating this:

+FR is running with speculative_token_map. Qwen2.5

[Swipe4057]

https://github.com/HArmonizedSS/HASS
324 -
speculative_num_steps-3
speculative_eagle_topk-2
speculative_num_draft_tokens-4

[keyboardAnt]

Acceptance

I've benchmarked acceptance to compare between FR-Spec and speculation without FR-Spec. The average number of accepted draft tokens:

• FR-Spec: 1.64 draft tokens
• w/o FR-Spec: 1.72 draft tokens

---

Details

Commands: Same as I used for the general benchmark detailed in my previous comment above.

FR-Spec:

python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \
    --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt \
    --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16

w/o FR-Spec: (remove the --speculative-token-map flag)

python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --speculative-algorithm EAGLE \
    --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \
    --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 \
    --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16

Code: The code is branched from v0.4.9.post4, and the only changes I made are for calculating and printing the acceptance: https://github.com/keyboardAnt/sglang/tree/nadav/acceptance

[eitanturok]

I re-ran the exact benchmarks you used and got different results @keyboardAnt: I found that fr-spec is faster than both speculative decoding and vanilla inference.

My results:

method	output throughput (tokens/sec)	command
vanilla	224.45 245.64 251.00	python3 -m sglang.bench_offline_throughput --model-path unsloth/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16
speculative decoding	241.43 288.53 305.02	python3 -m sglang.bench_offline_throughput --model-path unsloth/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16
fr-spec	302.19 310.59 310.53	python3 -m sglang.bench_offline_throughput --model-path unsloth/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16

To create the results I cloned the main branch of slgnag and rented an A100-80GB from Modal. I then ran each command 3 separate times and recorded the output throughput.

Why do we get different results? The truth is, I'm not sure. But here are some ideas:

1. Notice that speculative decoding outputs both 241.43 and 305.02 tokens/sec, a 26% relative increase in throughput. This means the variance is really high! Perhaps high variance could explain the discrepancy between our results.
2. Can not "warming up" the GPU properly be responsible for the difference between our results? After all, I ran the results 3 times and @keyboardAnt only ran it once. However, upon closer investigation the benchmark already contains a "warmup" stage. So I don't think that "not warming up" can explain the discrepancy in our results.

[keyboardAnt]

I re-ran the exact benchmarks you used and got different results @keyboardAnt: I found that fr-spec is faster than both speculative decoding and vanilla inference.

My results:

method output throughput (tokens/sec) command
vanilla 224.45
245.64
251.00   python3 -m sglang.bench_offline_throughput --model-path unsloth/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16
speculative decoding 241.43
288.53
305.02 python3 -m sglang.bench_offline_throughput --model-path unsloth/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16  
fr-spec 302.19
310.59
310.53 python3 -m sglang.bench_offline_throughput --model-path unsloth/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16  
To create the results I cloned the main branch of slgnag and rented an A100-80GB from Modal. I then ran each command 3 separate times and recorded the output throughput.

Why do we get different results? The truth is, I'm not sure. But here are some ideas:

1. Notice that speculative decoding outputs both 241.43 and 305.02 tokens/sec, a 26% relative increase in throughput. This means the variance is really high! Perhaps high variance could explain the discrepancy between our results.
2. Can not "warming up" the GPU properly be responsible for the difference between our results? After all, I ran the results 3 times and @keyboardAnt only ran it once. However, upon closer investigation the benchmark already contains a "warmup" stage. So I don't think that "not warming up" can explain the discrepancy in our results.

Thank you @eitanturok. Perhaps increasing --num-prompts from 10 to a larger number of prompts can reduce the variance that you mentioned.

[keyboardAnt]

Adding some results on main (commit 4d98e486499831f2b057b275dc9c2694b31d5e23):

Vanilla: (python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 1000):

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     1000      
Benchmark duration (s):                  32.61     
Total input tokens:                      296523    
Total generated tokens:                  186737    
Last generation throughput (tok/s):      226.89    
Request throughput (req/s):              30.66     
Input token throughput (tok/s):          9092.19   
Output token throughput (tok/s):         5725.86   
Total token throughput (tok/s):          14818.05  
==================================================

Speculative decoding: Seems to crash. for such a large number of input prompts, even after setting --mem-fraction-static 0.7, --cuda-graph-max-bs 2, and --num-prompts 10 (python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 1000 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction-static 0.7 --cuda-graph-max-bs 2).

---

python3 -m sglang.check_env:

Python: 3.12.11 (main, Jul 23 2025, 00:34:44) [Clang 20.1.4 ]
CUDA available: True
GPU 0: NVIDIA H100 NVL
GPU 0 Compute Capability: 9.0
CUDA_HOME: None
PyTorch: 2.8.0+cu128
sglang: 0.5.0rc2
sgl_kernel: 0.3.5
flashinfer_python: 0.2.11.post3
triton: 3.4.0
transformers: 4.55.2
torchao: 0.9.0
numpy: 2.3.2
aiohttp: 3.12.15
fastapi: 0.116.1
hf_transfer: 0.1.9
huggingface_hub: 0.34.4
interegular: 0.3.3
modelscope: 1.29.0
orjson: 3.11.2
outlines: 0.1.11
packaging: 25.0
psutil: 7.0.0
pydantic: 2.11.7
python-multipart: 0.0.20
pyzmq: 27.0.1
uvicorn: 0.35.0
uvloop: 0.21.0
vllm: Module Not Found
xgrammar: 0.1.22
openai: 1.99.1
tiktoken: 0.11.0
anthropic: 0.64.0
litellm: 1.75.8
decord: 0.6.0
NVIDIA Topology: 
        GPU0    NIC0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    0       0               N/A
NIC0    NODE     X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0


ulimit soft: 66000

[Edenzzzz]

Please attach the error

[keyboardAnt]

Please attach the error

@Edenzzzz, I'm rerunning on the latest commit (4c0bb411e):

Vanilla

python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 1000:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     1000      
Benchmark duration (s):                  59.19     
Total input tokens:                      296523    
Total generated tokens:                  186737    
Last generation throughput (tok/s):      99.94     
Request throughput (req/s):              16.90     
Input token throughput (tok/s):          5009.85   
Output token throughput (tok/s):         3154.98   
Total token throughput (tok/s):          8164.82   
==================================================

Speculative decoding

python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 1000 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction-static 0.7 --cuda-graph-max-bs 2:

[nadavt@ibdgxa01 sglang]$ python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 1000 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction-static 0.7 --cuda-graph-max-bs 2
W0818 17:43:15.268000 3755517 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
W0818 17:43:15.268000 3755517 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
[2025-08-18 17:43:19] Using default HuggingFace chat template with detected content format: string
W0818 17:43:31.180000 3757255 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
W0818 17:43:31.180000 3757255 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
W0818 17:43:31.285000 3757256 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
W0818 17:43:31.285000 3757256 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[2025-08-18 17:43:36] Attention backend not explicitly specified. Use flashinfer backend by default.
[2025-08-18 17:43:36] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2025-08-18 17:43:36] Init torch distributed ends. mem usage=0.00 GB
[2025-08-18 17:43:36] MOE_RUNNER_BACKEND is not initialized, using triton backend
[2025-08-18 17:43:37] Ignore import error when loading sglang.srt.models.glm4v_moe: No module named 'transformers.models.glm4v_moe'
[2025-08-18 17:43:37] Load weight begin. avail mem=78.84 GB
[2025-08-18 17:43:38] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:01<00:03,  1.15s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:02<00:02,  1.26s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:03<00:00,  1.07it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.08it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.01it/s]

[2025-08-18 17:43:43] Load weight end. type=LlamaForCausalLM, dtype=torch.bfloat16, avail mem=63.75 GB, mem usage=15.10 GB.
[2025-08-18 17:43:43] KV Cache is allocated. #tokens: 333532, K size: 20.36 GB, V size: 20.36 GB
[2025-08-18 17:43:43] Memory pool end. avail mem=22.93 GB
[2025-08-18 17:43:44] Capture cuda graph begin. This can take up to several minutes. avail mem=22.44 GB
[2025-08-18 17:43:44] Capture graph bs [1, 2]
Capturing batches (bs=2 avail_mem=22.30 GB):   0%|                                                  | 0/2 [00:00<?, ?it/s][2025-08-18 17:43:44] IS_TBO_ENABLED is not initialized, using False
Capturing batches (bs=1 avail_mem=22.14 GB): 100%|██████████████████████████████████████████| 2/2 [00:33<00:00, 16.58s/it]
[2025-08-18 17:44:17] Capture cuda graph end. Time elapsed: 33.54 s. mem usage=0.38 GB. avail mem=22.06 GB.
[2025-08-18 17:44:19] Warning: User-specified context_length (131072) is greater than the derived context_length (2048). This may lead to incorrect model outputs or CUDA errors.
[2025-08-18 17:44:19] Init torch distributed begin.
[2025-08-18 17:44:19] Init torch distributed ends. mem usage=0.00 GB
[2025-08-18 17:44:19] Load weight begin. avail mem=22.06 GB
[2025-08-18 17:44:20] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:04<00:00,  4.32s/it]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:04<00:00,  4.32s/it]

[2025-08-18 17:44:24] Load weight end. type=LlamaForCausalLMEagle, dtype=torch.float16, avail mem=19.63 GB, mem usage=2.43 GB.
[2025-08-18 17:44:24] KV Cache is allocated. #tokens: 333532, K size: 0.64 GB, V size: 0.64 GB
[2025-08-18 17:44:24] Memory pool end. avail mem=18.36 GB
[2025-08-18 17:44:26] Capture draft cuda graph begin. This can take up to several minutes. avail mem=20.01 GB
Capturing batches (bs=2 avail_mem=20.01 GB):   0%|                                                  | 0/2 [00:39<?, ?it/s]
[2025-08-18 17:45:05] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 121, in __init__
    self.capture()
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 149, in capture
    CudaGraphRunner.capture(self)
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/model_executor/graph_runner.py", line 492, in capture
    ) = self.capture_one_batch_size(bs, forward)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 262, in capture_one_batch_size
    run_once()
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 252, in run_once
    ret = self.eagle_worker.draft_forward(forward_batch)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/speculative/eagle_worker.py", line 647, in draft_forward
    logits_output, _ = self.draft_model_runner.forward(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/model_executor/model_runner.py", line 1741, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/model_executor/model_runner.py", line 1780, in _forward_raw
    ret = self.forward_decode(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/model_executor/model_runner.py", line 1663, in forward_decode
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/models/llama.py", line 465, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/models/llama_eagle.py", line 97, in forward
    hidden_states = self.fc(
                    ^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/.venv/lib/python3.12/site-packages/torch/nn/modules/linear.py", line 125, in forward
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: mat1 and mat2 must have the same dtype, but got Float and Half

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/managers/scheduler.py", line 2552, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/managers/scheduler.py", line 335, in __init__
    self.draft_worker = EAGLEWorker(
                        ^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/speculative/eagle_worker.py", line 168, in __init__
    self.init_cuda_graphs()
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/speculative/eagle_worker.py", line 291, in init_cuda_graphs
    self.cuda_graph_runner = EAGLEDraftCudaGraphRunner(self)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 123, in __init__
    raise Exception(
Exception: Capture cuda graph failed: mat1 and mat2 must have the same dtype, but got Float and Half
Possible solutions:
1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7)
2. set --cuda-graph-max-bs to a smaller value (e.g., 16)
3. disable torch compile by not using --enable-torch-compile
4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss)
Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose 


[2025-08-18 17:45:05] Received sigquit from a child process. It usually means the child failed.
Killed

python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 1000 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction-static 0.7 --cuda-graph-max-bs 2 --dtype float16 (added --dtype float16):

[nadavt@ibdgxa01 sglang]$ python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 1000 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction-static 0.7 --cuda-graph-max-bs 2 --dtype float16
W0818 17:46:49.227000 3782700 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
W0818 17:46:49.227000 3782700 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
[2025-08-18 17:46:51] Casting torch.bfloat16 to torch.float16.
[2025-08-18 17:46:53] Using default HuggingFace chat template with detected content format: string
W0818 17:47:04.744000 3784118 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
W0818 17:47:04.744000 3784118 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
W0818 17:47:05.313000 3784119 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
W0818 17:47:05.313000 3784119 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[2025-08-18 17:47:08] Casting torch.bfloat16 to torch.float16.
[2025-08-18 17:47:09] Casting torch.bfloat16 to torch.float16.
[2025-08-18 17:47:10] Attention backend not explicitly specified. Use flashinfer backend by default.
[2025-08-18 17:47:10] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2025-08-18 17:47:10] Init torch distributed ends. mem usage=0.00 GB
[2025-08-18 17:47:10] MOE_RUNNER_BACKEND is not initialized, using triton backend
[2025-08-18 17:47:10] Ignore import error when loading sglang.srt.models.glm4v_moe: No module named 'transformers.models.glm4v_moe'
[2025-08-18 17:47:11] Load weight begin. avail mem=78.84 GB
[2025-08-18 17:47:12] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:05<00:17,  5.89s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:11<00:12,  6.01s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:13<00:04,  4.03s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:19<00:00,  4.66s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:19<00:00,  4.82s/it]

[2025-08-18 17:47:32] Load weight end. type=LlamaForCausalLM, dtype=torch.float16, avail mem=63.75 GB, mem usage=15.10 GB.
[2025-08-18 17:47:32] KV Cache is allocated. #tokens: 333532, K size: 20.36 GB, V size: 20.36 GB
[2025-08-18 17:47:32] Memory pool end. avail mem=22.93 GB
[2025-08-18 17:47:32] Capture cuda graph begin. This can take up to several minutes. avail mem=22.44 GB
[2025-08-18 17:47:32] Capture graph bs [1, 2]
Capturing batches (bs=2 avail_mem=22.30 GB):   0%|                                                  | 0/2 [00:00<?, ?it/s][2025-08-18 17:47:33] IS_TBO_ENABLED is not initialized, using False
Capturing batches (bs=1 avail_mem=22.15 GB): 100%|██████████████████████████████████████████| 2/2 [01:53<00:00, 56.73s/it]
[2025-08-18 17:49:26] Capture cuda graph end. Time elapsed: 113.96 s. mem usage=0.38 GB. avail mem=22.06 GB.
[2025-08-18 17:49:28] Warning: User-specified context_length (131072) is greater than the derived context_length (2048). This may lead to incorrect model outputs or CUDA errors.
[2025-08-18 17:49:28] Init torch distributed begin.
[2025-08-18 17:49:28] Init torch distributed ends. mem usage=0.00 GB
[2025-08-18 17:49:28] Load weight begin. avail mem=22.06 GB
[2025-08-18 17:49:28] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.24s/it]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.24s/it]

[2025-08-18 17:49:30] Load weight end. type=LlamaForCausalLMEagle, dtype=torch.float16, avail mem=19.64 GB, mem usage=2.43 GB.
[2025-08-18 17:49:30] KV Cache is allocated. #tokens: 333532, K size: 0.64 GB, V size: 0.64 GB
[2025-08-18 17:49:30] Memory pool end. avail mem=18.36 GB
[2025-08-18 17:49:31] Capture draft cuda graph begin. This can take up to several minutes. avail mem=20.02 GB
Capturing batches (bs=1 avail_mem=19.92 GB): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:09<00:00,  4.73s/it]
[2025-08-18 17:49:41] Capture draft cuda graph end. Time elapsed: 9.81 s. mem usage=0.14 GB. avail mem=19.87 GB.
[2025-08-18 17:49:41] Capture draft extend cuda graph begin. This can take up to several minutes. avail mem=19.87 GB
Capturing batches (bs=1 avail_mem=19.85 GB): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 36.07it/s]
[2025-08-18 17:49:41] Capture draft extend cuda graph end. Time elapsed: 0.55 s. mem usage=0.04 GB. avail mem=19.83 GB.
[2025-08-18 17:49:41] max_total_num_tokens=333532, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=48, context_len=131072, available_gpu_mem=19.83 GB
#Input tokens: 296523
#Output tokens: 186737
#Input tokens: 4096
#Output tokens: 256
[2025-08-18 17:50:09] 
Warmup...
[2025-08-18 17:50:09] Prefill batch. #new-seq: 3, #new-token: 771, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-08-18 17:50:09] Prefill batch. #new-seq: 13, #new-token: 3315, #cached-token: 26, token usage: 0.00, #running-req: 3, #queue-req: 0, 
[2025-08-18 17:50:18] 
Benchmark...
[2025-08-18 17:50:18] Prefill batch. #new-seq: 8, #new-token: 1384, #cached-token: 8, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-08-18 17:50:18] Prefill batch. #new-seq: 23, #new-token: 8192, #cached-token: 26, token usage: 0.00, #running-req: 8, #queue-req: 52, 
[2025-08-18 17:50:19] length of new_indices: 22 != length of topk_p: 23, this should not happen
[2025-08-18 17:50:19] Prefill batch. #new-seq: 18, #new-token: 4225, #cached-token: 25, token usage: 0.03, #running-req: 30, #queue-req: 619, 
[2025-08-18 17:50:21] Prefill batch. #new-seq: 1, #new-token: 5, #cached-token: 2, token usage: 0.04, #running-req: 47, #queue-req: 951, 
[2025-08-18 17:50:22] Prefill batch. #new-seq: 1, #new-token: 285, #cached-token: 2, token usage: 0.04, #running-req: 47, #queue-req: 950, 
[2025-08-18 17:50:22] Prefill batch. #new-seq: 3, #new-token: 430, #cached-token: 4, token usage: 0.04, #running-req: 45, #queue-req: 947, 
[2025-08-18 17:50:23] Prefill batch. #new-seq: 1, #new-token: 527, #cached-token: 2, token usage: 0.04, #running-req: 47, #queue-req: 946, 
[2025-08-18 17:50:23] Prefill batch. #new-seq: 1, #new-token: 1972, #cached-token: 1, token usage: 0.04, #running-req: 47, #queue-req: 945, 
[2025-08-18 17:50:24] Prefill batch. #new-seq: 3, #new-token: 248, #cached-token: 5, token usage: 0.04, #running-req: 45, #queue-req: 942, 
[2025-08-18 17:50:24] Prefill batch. #new-seq: 3, #new-token: 605, #cached-token: 4, token usage: 0.04, #running-req: 45, #queue-req: 939, 
[2025-08-18 17:50:24] Prefill batch. #new-seq: 4, #new-token: 2037, #cached-token: 9, token usage: 0.04, #running-req: 44, #queue-req: 935, 
[2025-08-18 17:50:25] Prefill batch. #new-seq: 1, #new-token: 336, #cached-token: 2, token usage: 0.04, #running-req: 47, #queue-req: 934, 
[2025-08-18 17:50:26] Prefill batch. #new-seq: 2, #new-token: 334, #cached-token: 3, token usage: 0.04, #running-req: 46, #queue-req: 932, 
[2025-08-18 17:50:26] Prefill batch. #new-seq: 2, #new-token: 434, #cached-token: 6, token usage: 0.04, #running-req: 46, #queue-req: 930, 
[2025-08-18 17:50:26] Prefill batch. #new-seq: 1, #new-token: 238, #cached-token: 1, token usage: 0.04, #running-req: 47, #queue-req: 929, 
[2025-08-18 17:50:27] Prefill batch. #new-seq: 2, #new-token: 59, #cached-token: 2, token usage: 0.04, #running-req: 46, #queue-req: 927, 
[2025-08-18 17:50:28] Prefill batch. #new-seq: 1, #new-token: 17, #cached-token: 2, token usage: 0.04, #running-req: 47, #queue-req: 926, 
[2025-08-18 17:50:28] Prefill batch. #new-seq: 1, #new-token: 691, #cached-token: 3, token usage: 0.04, #running-req: 47, #queue-req: 925, 
[2025-08-18 17:50:29] Prefill batch. #new-seq: 1, #new-token: 679, #cached-token: 2, token usage: 0.05, #running-req: 47, #queue-req: 924, 
[2025-08-18 17:50:30] Prefill batch. #new-seq: 1, #new-token: 712, #cached-token: 1, token usage: 0.05, #running-req: 47, #queue-req: 923, 
[2025-08-18 17:50:30] Decode batch. #running-req: 47, #token: 16292, token usage: 0.05, accept len: 2.29, cuda graph: False, gen throughput (token/s): 74.52, #queue-req: 923, 
[2025-08-18 17:50:30] Prefill batch. #new-seq: 1, #new-token: 690, #cached-token: 1, token usage: 0.05, #running-req: 47, #queue-req: 922, 
[2025-08-18 17:50:31] Prefill batch. #new-seq: 1, #new-token: 390, #cached-token: 3, token usage: 0.05, #running-req: 47, #queue-req: 921, 
[2025-08-18 17:50:31] Prefill batch. #new-seq: 1, #new-token: 1214, #cached-token: 1, token usage: 0.05, #running-req: 47, #queue-req: 920, 
[2025-08-18 17:50:31] Prefill batch. #new-seq: 1, #new-token: 2895, #cached-token: 2, token usage: 0.06, #running-req: 47, #queue-req: 919, 
[2025-08-18 17:50:32] Prefill batch. #new-seq: 1, #new-token: 20, #cached-token: 1, token usage: 0.07, #running-req: 47, #queue-req: 918, 
[2025-08-18 17:50:33] Prefill batch. #new-seq: 2, #new-token: 1095, #cached-token: 10, token usage: 0.06, #running-req: 46, #queue-req: 916, 
[2025-08-18 17:50:34] Prefill batch. #new-seq: 1, #new-token: 575, #cached-token: 4, token usage: 0.07, #running-req: 47, #queue-req: 915, 
[2025-08-18 17:50:35] Prefill batch. #new-seq: 1, #new-token: 412, #cached-token: 2, token usage: 0.07, #running-req: 47, #queue-req: 914, 
[2025-08-18 17:50:36] Prefill batch. #new-seq: 1, #new-token: 4, #cached-token: 2, token usage: 0.07, #running-req: 47, #queue-req: 913, 
[2025-08-18 17:50:36] Prefill batch. #new-seq: 1, #new-token: 181, #cached-token: 1, token usage: 0.06, #running-req: 47, #queue-req: 912, 
[2025-08-18 17:50:36] Prefill batch. #new-seq: 2, #new-token: 586, #cached-token: 4, token usage: 0.07, #running-req: 46, #queue-req: 910, 
[2025-08-18 17:50:37] Prefill batch. #new-seq: 1, #new-token: 25, #cached-token: 1, token usage: 0.07, #running-req: 47, #queue-req: 909, 
[2025-08-18 17:50:38] Prefill batch. #new-seq: 3, #new-token: 878, #cached-token: 5, token usage: 0.06, #running-req: 45, #queue-req: 906, 
[2025-08-18 17:50:38] Prefill batch. #new-seq: 1, #new-token: 11, #cached-token: 1, token usage: 0.07, #running-req: 47, #queue-req: 905, 
[2025-08-18 17:50:38] Prefill batch. #new-seq: 1, #new-token: 564, #cached-token: 4, token usage: 0.07, #running-req: 47, #queue-req: 904, 
[2025-08-18 17:50:39] Prefill batch. #new-seq: 1, #new-token: 20, #cached-token: 3, token usage: 0.07, #running-req: 47, #queue-req: 903, 
[2025-08-18 17:50:41] Prefill batch. #new-seq: 1, #new-token: 108, #cached-token: 2, token usage: 0.06, #running-req: 47, #queue-req: 902, 
[2025-08-18 17:50:41] Decode batch. #running-req: 48, #token: 20503, token usage: 0.06, accept len: 2.70, cuda graph: False, gen throughput (token/s): 469.38, #queue-req: 902, 
[2025-08-18 17:50:41] Prefill batch. #new-seq: 2, #new-token: 320, #cached-token: 4, token usage: 0.06, #running-req: 46, #queue-req: 900, 
[2025-08-18 17:50:42] Prefill batch. #new-seq: 1, #new-token: 44, #cached-token: 3, token usage: 0.06, #running-req: 47, #queue-req: 899, 
[2025-08-18 17:50:43] Prefill batch. #new-seq: 1, #new-token: 304, #cached-token: 2, token usage: 0.06, #running-req: 47, #queue-req: 898, 
[2025-08-18 17:50:44] Prefill batch. #new-seq: 1, #new-token: 765, #cached-token: 3, token usage: 0.06, #running-req: 47, #queue-req: 897, 
[2025-08-18 17:50:44] Prefill batch. #new-seq: 1, #new-token: 116, #cached-token: 3, token usage: 0.07, #running-req: 47, #queue-req: 896, 
[2025-08-18 17:50:45] Prefill batch. #new-seq: 1, #new-token: 96, #cached-token: 1, token usage: 0.07, #running-req: 47, #queue-req: 895, 
[2025-08-18 17:50:45] Prefill batch. #new-seq: 1, #new-token: 5017, #cached-token: 1, token usage: 0.07, #running-req: 47, #queue-req: 894, 
[2025-08-18 17:50:47] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/managers/scheduler.py", line 2577, in run_scheduler_process
    scheduler.event_loop_normal()
  File "/home/projects/dharel/nadavt/repos/sglang/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/managers/scheduler.py", line 784, in event_loop_normal
    self.process_batch_result(batch, result)
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/managers/scheduler.py", line 1806, in process_batch_result
    self.process_batch_result_prefill(batch, result, launch_done)
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/managers/scheduler_output_processor_mixin.py", line 59, in process_batch_result_prefill
    next_token_ids = next_token_ids.tolist()
                     ^^^^^^^^^^^^^^^^^^^^^^^
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


[2025-08-18 17:50:47] Received sigquit from a child process. It usually means the child failed.
Killed

---

Environment

python3 -m sglang.check_env:

Python: 3.12.11 (main, Jul 23 2025, 00:34:44) [Clang 20.1.4 ]
CUDA available: True
GPU 0: NVIDIA A100-SXM4-80GB
GPU 0 Compute Capability: 8.0
CUDA_HOME: /apps/easybd/easybuild/amd/software/CUDA/12.6.0
NVCC: Cuda compilation tools, release 12.6, V12.6.20
CUDA Driver Version: 560.35.05
PyTorch: 2.8.0+cu128
sglang: 0.5.0rc2
sgl_kernel: 0.3.5
flashinfer_python: 0.2.11.post3
triton: 3.4.0
transformers: 4.55.2
torchao: 0.9.0
numpy: 2.3.2
aiohttp: 3.12.15
fastapi: 0.116.1
hf_transfer: 0.1.9
huggingface_hub: 0.34.4
interegular: 0.3.3
modelscope: 1.29.0
orjson: 3.11.2
outlines: 0.1.11
packaging: 25.0
psutil: 7.0.0
pydantic: 2.11.7
python-multipart: 0.0.20
pyzmq: 27.0.1
uvicorn: 0.35.0
uvloop: 0.21.0
vllm: Module Not Found
xgrammar: 0.1.22
openai: 1.99.1
tiktoken: 0.11.0
anthropic: 0.64.0
litellm: Module Not Found
decord: 0.6.0
NVIDIA Topology: 
        GPU0    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    NIC8    NIC9    NIC10   NIC11   CPU Affinity      NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     PXB     PXB     SYS     SYS             5N/A
NIC0    NODE     X      PXB     NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE
NIC1    NODE    PXB      X      NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE
NIC2    NODE    NODE    NODE     X      PXB     NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE
NIC3    NODE    NODE    NODE    PXB      X      NODE    NODE    NODE    NODE    NODE    NODE    NODE    NODE
NIC4    NODE    NODE    NODE    NODE    NODE     X      PIX     NODE    NODE    NODE    NODE    NODE    NODE
NIC5    NODE    NODE    NODE    NODE    NODE    PIX      X      NODE    NODE    NODE    NODE    NODE    NODE
NIC6    SYS     NODE    NODE    NODE    NODE    NODE    NODE     X      PXB     SYS     SYS     SYS     SYS
NIC7    SYS     NODE    NODE    NODE    NODE    NODE    NODE    PXB      X      SYS     SYS     SYS     SYS
NIC8    PXB     NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS      X      PXB     SYS     SYS
NIC9    PXB     NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     PXB      X      SYS     SYS
NIC10   SYS     NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      PIX
NIC11   SYS     NODE    NODE    NODE    NODE    NODE    NODE    SYS     SYS     SYS     SYS     PIX      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9
  NIC10: mlx5_10
  NIC11: mlx5_11


ulimit soft: 512000

[keyboardAnt]

tl;dr: Benchmarking the latest commit in main suggests that speculative decoding slows down the inference. From high to low average output token throughput:

1. Vanilla (345.07 tok/s)
2. FR-Spec (313.78 tok/s)
3. Eagle (290.88 tok/s)

---

I was able to run the benchmark for a lower number of prompts (over the same hardware and git commit mentioned in my last comment above), as follows:

Vanilla

python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16: (three repeats)

Arithmetic mean output token throughput (tok/s): 345.07

repeat 1:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  8.16      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      89.73     
Request throughput (req/s):              1.23      
Input token throughput (tok/s):          240.14    
Output token throughput (tok/s):         339.87    
Total token throughput (tok/s):          580.01    
==================================================

repeat 2:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  7.96      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      89.87     
Request throughput (req/s):              1.26      
Input token throughput (tok/s):          246.20    
Output token throughput (tok/s):         348.45    
Total token throughput (tok/s):          594.64    
==================================================

repeat 3:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  8.00      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      89.72     
Request throughput (req/s):              1.25      
Input token throughput (tok/s):          245.10    
Output token throughput (tok/s):         346.89    
Total token throughput (tok/s):          591.98    
==================================================

Speculative decoding (Eagle)

python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --speculative-algorithm EAGLE \ --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \ --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 \ --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16: (three repeats)

Arithmetic mean output token throughput (tok/s): 290.88

repeat 1:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  10.77     
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      127.16    
Request throughput (req/s):              0.93      
Input token throughput (tok/s):          181.94    
Output token throughput (tok/s):         257.50    
Total token throughput (tok/s):          439.44    
==================================================

repeat 2:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  8.59      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      156.17    
Request throughput (req/s):              1.16      
Input token throughput (tok/s):          228.19    
Output token throughput (tok/s):         322.95    
Total token throughput (tok/s):          551.14    
==================================================

repeat 3:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  9.49      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      133.89    
Request throughput (req/s):              1.05      
Input token throughput (tok/s):          206.44    
Output token throughput (tok/s):         292.18    
Total token throughput (tok/s):          498.62    
==================================================

However, benchmarking speculative decoding for 100 prompts (python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 100 --speculative-algorithm EAGLE --speculative-draft-model-path lms ys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16) fails with the following exception:

W0818 18:05:52.238000 3924290 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
W0818 18:05:52.238000 3924290 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
WARNING:sglang.srt.server_args:Overlap scheduler is disabled because of using eagle speculative decoding.
[2025-08-18 18:05:54] Casting torch.bfloat16 to torch.float16.
[2025-08-18 18:05:56] Using default HuggingFace chat template with detected content format: string
W0818 18:06:11.678000 3926020 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
W0818 18:06:11.678000 3926020 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
W0818 18:06:11.681000 3926019 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
W0818 18:06:11.681000 3926019 .venv/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[2025-08-18 18:06:15] Casting torch.bfloat16 to torch.float16.
[2025-08-18 18:06:17] Casting torch.bfloat16 to torch.float16.
[2025-08-18 18:06:17] Attention backend not explicitly specified. Use flashinfer backend by default.
[2025-08-18 18:06:17] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2025-08-18 18:06:18] Init torch distributed ends. mem usage=0.00 GB
[2025-08-18 18:06:18] MOE_RUNNER_BACKEND is not initialized, using triton backend
[2025-08-18 18:06:18] Ignore import error when loading sglang.srt.models.glm4v_moe: No module named 'transformers.models.glm4v_moe'
[2025-08-18 18:06:18] Load weight begin. avail mem=78.84 GB
[2025-08-18 18:06:19] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:06<00:20,  6.86s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:13<00:14,  7.01s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:15<00:04,  4.71s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:22<00:00,  5.46s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:22<00:00,  5.64s/it]

[2025-08-18 18:06:43] Load weight end. type=LlamaForCausalLM, dtype=torch.float16, avail mem=63.75 GB, mem usage=15.10 GB.
[2025-08-18 18:06:43] KV Cache is allocated. #tokens: 333532, K size: 20.36 GB, V size: 20.36 GB
[2025-08-18 18:06:43] Memory pool end. avail mem=22.93 GB
[2025-08-18 18:06:43] Capture cuda graph begin. This can take up to several minutes. avail mem=22.44 GB
[2025-08-18 18:06:43] Capture graph bs [1, 2]
Capturing batches (bs=2 avail_mem=22.30 GB):   0%|                                                                                                                                      | 0/2 [00:00<?, ?it/s][2025-08-18 18:06:44] IS_TBO_ENABLED is not initialized, using False
Capturing batches (bs=1 avail_mem=22.15 GB): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  2.00it/s]
[2025-08-18 18:06:45] Capture cuda graph end. Time elapsed: 1.60 s. mem usage=0.38 GB. avail mem=22.06 GB.
[2025-08-18 18:06:47] Warning: User-specified context_length (131072) is greater than the derived context_length (2048). This may lead to incorrect model outputs or CUDA errors.
[2025-08-18 18:06:47] Init torch distributed begin.
[2025-08-18 18:06:47] Init torch distributed ends. mem usage=0.00 GB
[2025-08-18 18:06:47] Load weight begin. avail mem=22.06 GB
[2025-08-18 18:06:47] Using model weights format ['*.bin']
Loading pt checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.62s/it]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:01<00:00,  1.62s/it]

[2025-08-18 18:06:49] Load weight end. type=LlamaForCausalLMEagle, dtype=torch.float16, avail mem=19.64 GB, mem usage=2.43 GB.
[2025-08-18 18:06:49] KV Cache is allocated. #tokens: 333532, K size: 0.64 GB, V size: 0.64 GB
[2025-08-18 18:06:49] Memory pool end. avail mem=18.36 GB
[2025-08-18 18:06:51] Capture draft cuda graph begin. This can take up to several minutes. avail mem=20.02 GB
Capturing batches (bs=1 avail_mem=19.92 GB): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:06<00:00,  3.31s/it]
[2025-08-18 18:06:58] Capture draft cuda graph end. Time elapsed: 7.07 s. mem usage=0.14 GB. avail mem=19.87 GB.
[2025-08-18 18:06:58] Capture draft extend cuda graph begin. This can take up to several minutes. avail mem=19.87 GB
Capturing batches (bs=1 avail_mem=19.85 GB): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 41.23it/s]
[2025-08-18 18:06:59] Capture draft extend cuda graph end. Time elapsed: 0.54 s. mem usage=0.04 GB. avail mem=19.83 GB.
[2025-08-18 18:06:59] max_total_num_tokens=333532, chunked_prefill_size=8192, max_prefill_tokens=16384, max_running_requests=48, context_len=131072, available_gpu_mem=19.83 GB
#Input tokens: 33559
#Output tokens: 21265
#Input tokens: 4096
#Output tokens: 256
[2025-08-18 18:07:28] 
Warmup...
[2025-08-18 18:07:28] Prefill batch. #new-seq: 1, #new-token: 257, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-08-18 18:07:28] Prefill batch. #new-seq: 15, #new-token: 3825, #cached-token: 30, token usage: 0.00, #running-req: 1, #queue-req: 0, 
[2025-08-18 18:07:31] 
Benchmark...
[2025-08-18 18:07:31] Prefill batch. #new-seq: 2, #new-token: 44, #cached-token: 2, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-08-18 18:07:31] Prefill batch. #new-seq: 13, #new-token: 5284, #cached-token: 13, token usage: 0.00, #running-req: 2, #queue-req: 0, 
[2025-08-18 18:07:32] Prefill batch. #new-seq: 33, #new-token: 8192, #cached-token: 39, token usage: 0.02, #running-req: 15, #queue-req: 52, 
[2025-08-18 18:07:32] length of new_indices: 32 != length of topk_p: 33, this should not happen
[2025-08-18 18:07:32] Prefill batch. #new-seq: 1, #new-token: 286, #cached-token: 0, token usage: 0.04, #running-req: 47, #queue-req: 52, 
[2025-08-18 18:07:33] Prefill batch. #new-seq: 1, #new-token: 5, #cached-token: 2, token usage: 0.04, #running-req: 47, #queue-req: 51, 
[2025-08-18 18:07:33] Prefill batch. #new-seq: 1, #new-token: 285, #cached-token: 2, token usage: 0.04, #running-req: 47, #queue-req: 50, 
[2025-08-18 18:07:33] Prefill batch. #new-seq: 3, #new-token: 430, #cached-token: 4, token usage: 0.04, #running-req: 45, #queue-req: 47, 
[2025-08-18 18:07:34] Prefill batch. #new-seq: 1, #new-token: 527, #cached-token: 2, token usage: 0.04, #running-req: 47, #queue-req: 46, 
[2025-08-18 18:07:34] Prefill batch. #new-seq: 1, #new-token: 1972, #cached-token: 1, token usage: 0.04, #running-req: 47, #queue-req: 45, 
[2025-08-18 18:07:35] Prefill batch. #new-seq: 3, #new-token: 248, #cached-token: 5, token usage: 0.04, #running-req: 45, #queue-req: 42, 
[2025-08-18 18:07:35] Prefill batch. #new-seq: 3, #new-token: 605, #cached-token: 4, token usage: 0.04, #running-req: 45, #queue-req: 39, 
[2025-08-18 18:07:35] Prefill batch. #new-seq: 4, #new-token: 2037, #cached-token: 9, token usage: 0.04, #running-req: 44, #queue-req: 35, 
[2025-08-18 18:07:36] Prefill batch. #new-seq: 1, #new-token: 336, #cached-token: 2, token usage: 0.04, #running-req: 47, #queue-req: 34, 
[2025-08-18 18:07:36] Prefill batch. #new-seq: 2, #new-token: 334, #cached-token: 3, token usage: 0.04, #running-req: 46, #queue-req: 32, 
[2025-08-18 18:07:37] Prefill batch. #new-seq: 2, #new-token: 434, #cached-token: 6, token usage: 0.04, #running-req: 46, #queue-req: 30, 
[2025-08-18 18:07:37] Prefill batch. #new-seq: 1, #new-token: 238, #cached-token: 1, token usage: 0.04, #running-req: 47, #queue-req: 29, 
[2025-08-18 18:07:38] Prefill batch. #new-seq: 2, #new-token: 59, #cached-token: 2, token usage: 0.04, #running-req: 46, #queue-req: 27, 
[2025-08-18 18:07:38] Prefill batch. #new-seq: 1, #new-token: 17, #cached-token: 2, token usage: 0.04, #running-req: 47, #queue-req: 26, 
[2025-08-18 18:07:39] Prefill batch. #new-seq: 1, #new-token: 691, #cached-token: 3, token usage: 0.04, #running-req: 47, #queue-req: 25, 
[2025-08-18 18:07:39] Prefill batch. #new-seq: 1, #new-token: 679, #cached-token: 2, token usage: 0.05, #running-req: 47, #queue-req: 24, 
[2025-08-18 18:07:40] Prefill batch. #new-seq: 1, #new-token: 712, #cached-token: 1, token usage: 0.05, #running-req: 47, #queue-req: 23, 
[2025-08-18 18:07:41] Decode batch. #running-req: 48, #token: 16285, token usage: 0.05, accept len: 2.28, cuda graph: False, gen throughput (token/s): 83.39, #queue-req: 23, 
[2025-08-18 18:07:41] Prefill batch. #new-seq: 1, #new-token: 690, #cached-token: 1, token usage: 0.05, #running-req: 47, #queue-req: 22, 
[2025-08-18 18:07:41] Prefill batch. #new-seq: 1, #new-token: 390, #cached-token: 3, token usage: 0.05, #running-req: 47, #queue-req: 21, 
[2025-08-18 18:07:42] Prefill batch. #new-seq: 1, #new-token: 1214, #cached-token: 1, token usage: 0.05, #running-req: 47, #queue-req: 20, 
[2025-08-18 18:07:42] Prefill batch. #new-seq: 1, #new-token: 2895, #cached-token: 2, token usage: 0.06, #running-req: 47, #queue-req: 19, 
[2025-08-18 18:07:43] Prefill batch. #new-seq: 1, #new-token: 20, #cached-token: 1, token usage: 0.07, #running-req: 47, #queue-req: 18, 
[2025-08-18 18:07:44] Prefill batch. #new-seq: 2, #new-token: 1095, #cached-token: 10, token usage: 0.06, #running-req: 46, #queue-req: 16, 
[2025-08-18 18:07:45] Prefill batch. #new-seq: 1, #new-token: 575, #cached-token: 4, token usage: 0.07, #running-req: 47, #queue-req: 15, 
[2025-08-18 18:07:45] Prefill batch. #new-seq: 1, #new-token: 412, #cached-token: 2, token usage: 0.07, #running-req: 47, #queue-req: 14, 
[2025-08-18 18:07:46] Prefill batch. #new-seq: 1, #new-token: 4, #cached-token: 2, token usage: 0.07, #running-req: 47, #queue-req: 13, 
[2025-08-18 18:07:47] Prefill batch. #new-seq: 1, #new-token: 181, #cached-token: 1, token usage: 0.06, #running-req: 47, #queue-req: 12, 
[2025-08-18 18:07:47] Prefill batch. #new-seq: 2, #new-token: 586, #cached-token: 4, token usage: 0.07, #running-req: 46, #queue-req: 10, 
[2025-08-18 18:07:48] Prefill batch. #new-seq: 1, #new-token: 25, #cached-token: 1, token usage: 0.07, #running-req: 47, #queue-req: 9, 
[2025-08-18 18:07:48] Prefill batch. #new-seq: 1, #new-token: 39, #cached-token: 1, token usage: 0.06, #running-req: 47, #queue-req: 8, 
[2025-08-18 18:07:49] Prefill batch. #new-seq: 2, #new-token: 839, #cached-token: 4, token usage: 0.06, #running-req: 46, #queue-req: 6, 
[2025-08-18 18:07:49] Prefill batch. #new-seq: 1, #new-token: 11, #cached-token: 1, token usage: 0.07, #running-req: 47, #queue-req: 5, 
[2025-08-18 18:07:49] Prefill batch. #new-seq: 1, #new-token: 564, #cached-token: 4, token usage: 0.07, #running-req: 47, #queue-req: 4, 
[2025-08-18 18:07:50] Prefill batch. #new-seq: 1, #new-token: 20, #cached-token: 3, token usage: 0.07, #running-req: 47, #queue-req: 3, 
[2025-08-18 18:07:52] Prefill batch. #new-seq: 1, #new-token: 108, #cached-token: 2, token usage: 0.06, #running-req: 47, #queue-req: 2, 
[2025-08-18 18:07:52] Decode batch. #running-req: 48, #token: 20847, token usage: 0.06, accept len: 2.70, cuda graph: False, gen throughput (token/s): 460.07, #queue-req: 2, 
[2025-08-18 18:07:52] Prefill batch. #new-seq: 1, #new-token: 23, #cached-token: 3, token usage: 0.06, #running-req: 47, #queue-req: 1, 
[2025-08-18 18:07:52] Prefill batch. #new-seq: 1, #new-token: 297, #cached-token: 1, token usage: 0.06, #running-req: 47, #queue-req: 0, 
[2025-08-18 18:08:05] Decode batch. #running-req: 31, #token: 18696, token usage: 0.06, accept len: 2.86, cuda graph: False, gen throughput (token/s): 369.32, #queue-req: 0, 
[2025-08-18 18:08:13] Decode batch. #running-req: 18, #token: 15125, token usage: 0.05, accept len: 3.06, cuda graph: False, gen throughput (token/s): 346.84, #queue-req: 0, 
[2025-08-18 18:08:18] Decode batch. #running-req: 14, #token: 15347, token usage: 0.05, accept len: 2.94, cuda graph: False, gen throughput (token/s): 395.69, #queue-req: 0, 
[2025-08-18 18:08:21] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/managers/scheduler.py", line 2577, in run_scheduler_process
    scheduler.event_loop_normal()
  File "/home/projects/dharel/nadavt/repos/sglang/.venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/managers/scheduler.py", line 783, in event_loop_normal
    result = self.run_batch(batch)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/managers/scheduler.py", line 1753, in run_batch
    ) = self.draft_worker.forward_batch_speculative_generation(batch)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/speculative/eagle_worker.py", line 343, in forward_batch_speculative_generation
    self.verify(batch, spec_info)
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/speculative/eagle_worker.py", line 683, in verify
    self.target_worker.forward_batch_generation(
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/managers/tp_worker.py", line 238, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/model_executor/model_runner.py", line 1741, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/model_executor/model_runner.py", line 1786, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/model_executor/model_runner.py", line 1677, in forward_extend
    self.attn_backend.init_forward_metadata(forward_batch)
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/layers/attention/flashinfer_backend.py", line 236, in init_forward_metadata
    self.indices_updater_prefill.update(
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/layers/attention/flashinfer_backend.py", line 882, in update_single_wrapper
    self.call_begin_forward(
  File "/home/projects/dharel/nadavt/repos/sglang/python/sglang/srt/layers/attention/flashinfer_backend.py", line 1054, in call_begin_forward
    wrapper_paged.begin_forward(
  File "/home/projects/dharel/nadavt/repos/sglang/.venv/lib/python3.12/site-packages/flashinfer/prefill.py", line 1658, in plan
    mask_indptr = _compute_page_mask_indptr(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/projects/dharel/nadavt/repos/sglang/.venv/lib/python3.12/site-packages/flashinfer/prefill.py", line 1193, in _compute_page_mask_indptr
    mask_indptr[0] = 0
    ~~~~~~~~~~~^^^
torch.AcceleratorError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


[2025-08-18 18:08:21] Received sigquit from a child process. It usually means the child failed.
Killed

FR-Spec

Adding the --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt argument at the end
(python3 -m sglang.bench_offline_throughput --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --speculative-algorithm EAGLE \ --speculative-draft-model-path lmsys/sglang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 \ --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 \ --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt): (three repeats)

Arithmetic mean output token throughput (tok/s): 313.78

repeat 1:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  8.96      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      150.74    
Request throughput (req/s):              1.12      
Input token throughput (tok/s):          218.63    
Output token throughput (tok/s):         309.43    
Total token throughput (tok/s):          528.07    
==================================================

repeat 2:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  8.64      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      160.75    
Request throughput (req/s):              1.16      
Input token throughput (tok/s):          226.82    
Output token throughput (tok/s):         321.02    
Total token throughput (tok/s):          547.85    
==================================================

repeat 3:

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  8.92      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      153.99    
Request throughput (req/s):              1.12      
Input token throughput (tok/s):          219.66    
Output token throughput (tok/s):         310.88    
Total token throughput (tok/s):          530.54    
==================================================

[eitanturok]

@keyboardAnt You reported the average output throughput for vanilla inference is 345.07 tok/s while I ran the same exact command and got 240.36 tok/s. This is very strange.

Any ideas on what the difference might be? Different hardware? Different environment? They seem to both be on A100-80GB and using the main branch of sglang. Unfortunately, I didn't save my environment when I ran the results so we can't know precisely more than that...

[keyboardAnt]

@keyboardAnt You reported the average output throughput for vanilla inference is 345.07 tok/s while I ran the same exact command and got 240.36 tok/s. This is very strange.

Any ideas on what the difference might be? Different hardware? Different environment? They seem to both be on A100-80GB and using the main branch of sglang. Unfortunately, I didn't save my environment when I ran the results so we can't know precisely more than that...

1. We must have used different git commits because I used a git commit that was created after you posted your comment.
2. I see you used a different model (unsloth/...), although unsloth's core maintainer claims that it is exactly the same model.

As for 2, I reran FR-Spec over the unsloth model and got a similar result: (python3 -m sglang.bench_offline_throughput --model-path unsloth/Meta-Llama-3.1-8B-Instruct --num-prompts 10 --speculative-algorithm EAGLE --speculative-draft-model-path lmsys/s glang-EAGLE-LLaMA3-Instruct-8B --speculative-num-steps 5 --speculative-eagle-topk 8 --speculative-num-draft-tokens 64 --mem-fraction 0.7 --cuda-graph-max-bs 2 --dtype float16 --speculative-token-map thunlp/LLaMA3-Instruct-8B-FR-Spec/freq_32768.pt)

====== Offline Throughput Benchmark Result =======
Backend:                                 engine    
Successful requests:                     10        
Benchmark duration (s):                  8.92      
Total input tokens:                      1960      
Total generated tokens:                  2774      
Last generation throughput (tok/s):      154.52    
Request throughput (req/s):              1.12      
Input token throughput (tok/s):          219.65    
Output token throughput (tok/s):         310.87    
Total token throughput (tok/s):          530.52    
==================================================

This is on the same hardware and env as before.