How can we reproduce the fact that using kvcached is 2-28 times faster than not using kvcached in TTFT?

[yugenlgy]

Where are the service startup and benchmark scripts located?

GPU =80G-H800 ，model=/test1/weights/Qwen3-8B
I'm using the bench_latency_benefit folder to start and test the service.
with kvcached
python controller/launch.py --config benchmarks/bench_latency_benefit/bench-config-vllm-llama-8b-with-kvcached.yaml

kvcached: # kvcached environment variables for all instances
kvcached_gpu_utilization: 0.95
kvcached_page_prealloc_enabled: true
kvcached_min_reserved_pages: 5
kvcached_max_reserved_pages: 10
kvcached_sanity_check: false
kvcached_log_level: INFO

router: # router configuration
enable_router: true
router_port: 8000
router_host: localhost

sleep_manager: # sleep configuration
idle_threshold_seconds: 300 # 5 minutes - time before putting idle models to sleep
check_interval_seconds: 60 # Check every minute for idle models
auto_sleep_enabled: false # Enable automatic sleep mode for idle models
wakeup_on_request: true # Automatically wake models when requests come in
min_sleep_duration: 80 # Minimum time to keep model asleep (2 minutes)

#launch_delay_seconds: 30 # Delay between launching each instance

instances: # instances configuration

• name: instance1
  model: /test1/weights/Qwen3-8B
  engine: vllm
  using_venv: true
  venv_path: /workspace/kvcached/engine_integration/vllm-source-venv
  kvcached_env: # kvcached environment variables for this instance

  • "ENABLE_KVCACHED=true"
  • "KVCACHED_AUTOPATCH=1"
    engine_env:
  • "VLLM_USE_V1=1"
  • "VLLM_ATTENTION_BACKEND=FLASH_ATTN"
  • "VLLM_SERVER_DEV_MODE=1" # To enable model sleep
    engine_args:
  • "--disable-log-requests"
  • "--no-enable-prefix-caching"
  • "--host=localhost"
  • "--port=12346"

  - "--gpu-memory-utilization 0.31"

  • "--max-model-len 2048"

• name: instance2
  #model: /test1/weights/DeepSeek-R1-Distill-Qwen-1.5B
  model: /test1/weights/Qwen3-8B
  engine: vllm
  using_venv: true
  venv_path: /workspace/kvcached/engine_integration/vllm-source-venv
  kvcached_env: # kvcached environment variables for this instance

  • "ENABLE_KVCACHED=true"
  • "KVCACHED_AUTOPATCH=1"
    engine_env:
  • "VLLM_USE_V1=1"
  • "VLLM_ATTENTION_BACKEND=FLASH_ATTN"
  • "VLLM_SERVER_DEV_MODE=1" # To enable model sleep
    engine_args:
  • "--disable-log-requests"
  • "--no-enable-prefix-caching"
  • "--host=localhost"
  • "--port=30000"

  - "--gpu-memory-utilization 0.31"

  • "--max-model-len 2048"

• name: instance3
  #model: /test1/weights/DeepSeek-R1-Distill-Qwen-1.5B
  model: /test1/weights/Qwen3-8B
  engine: vllm
  using_venv: true
  venv_path: /workspace/kvcached/engine_integration/vllm-source-venv
  kvcached_env: # kvcached environment variables for this instance

  • "ENABLE_KVCACHED=true"
  • "KVCACHED_AUTOPATCH=1"
    engine_env:
  • "VLLM_USE_V1=1"
  • "VLLM_ATTENTION_BACKEND=FLASH_ATTN"
  • "VLLM_SERVER_DEV_MODE=1" # To enable model sleep
    engine_args:
  • "--disable-log-requests"
  • "--no-enable-prefix-caching"
  • "--host=localhost"
  • "--port=40000"

  - "--gpu-memory-utilization 0.31"

  • "--max-model-len 2048"

without kvcached
source /workspace/kvcached/engine_integration/vllm-source-venv/bin/activate
export VLLM_USE_V1=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export ENABLE_KVCACHED=false
export KVCACHED_AUTOPATCH=0

CUDA_VISIBLE_DEVICES=7
vllm serve /test1/weights/Qwen3-8B
--disable-log-requests
--no-enable-prefix-caching
--host=localhost
--port=22347
--gpu-memory-utilization 0.3
--tensor-parallel-size 1
--max-model-len 2028

source /workspace/kvcached/engine_integration/vllm-source-venv/bin/activate
export VLLM_USE_V1=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export ENABLE_KVCACHED=false
export KVCACHED_AUTOPATCH=0

CUDA_VISIBLE_DEVICES=7
vllm serve /test1/weights/Qwen3-8B
--disable-log-requests
--no-enable-prefix-caching
--host=localhost
--port=22348
--gpu-memory-utilization 0.3
--tensor-parallel-size 1
--max-model-len 2028

source /workspace/kvcached/engine_integration/vllm-source-venv/bin/activate
export VLLM_USE_V1=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export ENABLE_KVCACHED=false
export KVCACHED_AUTOPATCH=0

CUDA_VISIBLE_DEVICES=7
vllm serve /test1/weights/Qwen3-8B
--disable-log-requests
--no-enable-prefix-caching
--host=localhost
--port=22349
--gpu-memory-utilization 0.3
--tensor-parallel-size 1
--max-model-len 2048

I set --max-model-len 2048 to ensure the smooth startup of all three models.

benchmark：bash benchmarks/bench_latency_benefit/run_benchmark.sh
I'm using benchmarks/bench_latency_benefit/run_benchmark.sh for benchmark testing, but all the results show that TTFT without kvcached is faster than with kvcached.

My requirement is to reproduce the 2-28x speedup of TTFT.

[yugenlgy]

@jiarong0907 Hello, how can I reproduce TTFT faster 2-28?

[jiarong0907]

@yugenlgy Do you mean the performance figure in the readme? If so, @cui36 can comment on this.

[yugenlgy]

Use kvcached to start three DeepSeek-R1-Distill-Llama-8B model scriptskvcached: # kvcached environment variables for all instances

kvcached_gpu_utilization: 0.95
kvcached_page_prealloc_enabled: true
#kvcached_min_reserved_pages: 10
#kvcached_max_reserved_pages: 30
kvcached_min_reserved_pages: 5
kvcached_max_reserved_pages: 10
kvcached_sanity_check: false
kvcached_log_level: INFO

router: # router configuration
enable_router: true
router_port: 8000
router_host: localhost

#sleep_manager: # sleep configuration
#idle_threshold_seconds: 300 # 5 minutes - time before putting idle models to sleep

check_interval_seconds: 60 # Check every minute for idle models

auto_sleep_enabled: false # Enable automatic sleep mode for idle models

wakeup_on_request: true # Automatically wake models when requests come in

min_sleep_duration: 80 # Minimum time to keep model asleep (2 minutes)

launch_delay_seconds: 30 # Delay between launching each instance

instances: # instances configuration

• name: instance1
  model: /test1/weights/DeepSeek-R1-Distill-Llama-8B
  engine: vllm
  using_venv: true
  venv_path: /workspace/kvcached/engine_integration/vllm-source-venv
  kvcached_env: # kvcached environment variables for this instance

  • "ENABLE_KVCACHED=true"
  • "KVCACHED_AUTOPATCH=1"
    engine_env:
  • "VLLM_USE_V1=1"
  • "VLLM_ATTENTION_BACKEND=FLASH_ATTN"
    #- "VLLM_SERVER_DEV_MODE=1" # To enable model sleep
  • "CUDA_VISIBLE_DEVICES=7"
    engine_args:
  • "--disable-log-requests"
  • "--no-enable-prefix-caching"
  • "--host=localhost"
  • "--port=12346"
    #- "--gpu-memory-utilization 0.3"
  • "--enable-sleep-mode"
  • "--max-model-len 62000"
    #- "--max-model-len 8096"

• name: instance2
  model: /test1/weights/DeepSeek-R1-Distill-Llama-8B
  engine: vllm
  using_venv: true
  venv_path: /workspace/kvcached/engine_integration/vllm-source-venv
  kvcached_env: # kvcached environment variables for this instance

  • "ENABLE_KVCACHED=true"
  • "KVCACHED_AUTOPATCH=1"
    engine_env:
  • "VLLM_USE_V1=1"
  • "VLLM_ATTENTION_BACKEND=FLASH_ATTN"
    #- "VLLM_SERVER_DEV_MODE=1" # To enable model sleep
  • "CUDA_VISIBLE_DEVICES=7"
    engine_args:
  • "--disable-log-requests"
  • "--no-enable-prefix-caching"
  • "--host=localhost"
  • "--port=30000"
    #- "--gpu-memory-utilization 0.3"
  • "--enable-sleep-mode"
  • "--max-model-len 62000"
    #- "--max-model-len 8096"

  model: /test1/weights/DeepSeek-R1-Distill-Llama-8B
  engine: vllm
  using_venv: true
  venv_path: /workspace/kvcached/engine_integration/vllm-source-venv
  kvcached_env: # kvcached environment variables for this instance

  • "ENABLE_KVCACHED=true"
  • "KVCACHED_AUTOPATCH=1"
    engine_env:
  • "VLLM_USE_V1=1"
  • "VLLM_ATTENTION_BACKEND=FLASH_ATTN"
    #- "VLLM_SERVER_DEV_MODE=1" # To enable model sleep
  • "CUDA_VISIBLE_DEVICES=7"
    engine_args:
  • "--disable-log-requests"
  • "--no-enable-prefix-caching"
  • "--host=localhost"
  • "--port=40000"
    #- "--gpu-memory-utilization 0.3"
  • "--enable-sleep-mode"
  • "--max-model-len 62000"
    #- "--max-model-len 8096"

Scripts for launching three DeepSeek-R1-Distill-Llama-8B models without using kvcached：

source /workspace/kvcached/engine_integration/vllm-source-venv/bin/activate
export VLLM_USE_V1=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export ENABLE_KVCACHED=false
export KVCACHED_AUTOPATCH=0

CUDA_VISIBLE_DEVICES=7
vllm serve /test1/weights/DeepSeek-R1-Distill-Llama-8B
--disable-log-requests
--no-enable-prefix-caching
--host=localhost
--port=12346
--gpu-memory-utilization 0.3
--tensor-parallel-size 1
--max-model-len 8096

source /workspace/kvcached/engine_integration/vllm-source-venv/bin/activate
export VLLM_USE_V1=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export ENABLE_KVCACHED=false
export KVCACHED_AUTOPATCH=0

CUDA_VISIBLE_DEVICES=7
vllm serve /test1/weights/DeepSeek-R1-Distill-Llama-8B
--disable-log-requests
--no-enable-prefix-caching
--host=localhost
--port=30000
--gpu-memory-utilization 0.3
--tensor-parallel-size 1
--max-model-len 8096

source /workspace/kvcached/engine_integration/vllm-source-venv/bin/activate
export VLLM_USE_V1=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export ENABLE_KVCACHED=false
export KVCACHED_AUTOPATCH=0

CUDA_VISIBLE_DEVICES=7
vllm serve /test1/weights/DeepSeek-R1-Distill-Llama-8B
--disable-log-requests
--no-enable-prefix-caching
--host=localhost
--port=40000
--gpu-memory-utilization 0.3
--tensor-parallel-size 1
--max-model-len 8096

source /workspace/kvcached/engine_integration/vllm-source-venv/bin/activate
export VLLM_USE_V1=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN
export ENABLE_KVCACHED=false
export KVCACHED_AUTOPATCH=0

CUDA_VISIBLE_DEVICES=7
vllm serve /test1/weights/DeepSeek-R1-Distill-Llama-8B
--disable-log-requests
--no-enable-prefix-caching
--host=localhost
--port=40000
--gpu-memory-utilization 0.3
--tensor-parallel-size 1
--max-model-len 2048

Test benchmark script：bash benchmarks/bench_latency_benefit/run_benchmark.sh 17 256

without kvcached result

============ Serving Benchmark Result ============
Successful requests: 479
Benchmark duration (s): 225.88
Total input tokens: 122624
Total generated tokens: 122624
Request throughput (req/s): 2.12
Output token throughput (tok/s): 542.87
Total Token throughput (tok/s): 1085.74
---------------Time to First Token----------------
Mean TTFT (ms): 40.95
Median TTFT (ms): 37.93
P99 TTFT (ms): 115.41
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 17.46
Median TPOT (ms): 17.06
P99 TPOT (ms): 24.25
---------------Inter-token Latency----------------
Mean ITL (ms): 17.47
Median ITL (ms): 15.80
P99 ITL (ms): 26.32
----------------End-to-end Latency----------------
Mean E2EL (ms): 4492.98
Median E2EL (ms): 4392.20
P99 E2EL (ms): 6233.15

with kvcached result result
Successful requests: 479
Benchmark duration (s): 227.37
Total input tokens: 122624
Total generated tokens: 122624
Request throughput (req/s): 2.11
Output token throughput (tok/s): 539.32
Total Token throughput (tok/s): 1078.65
---------------Time to First Token----------------
Mean TTFT (ms): 3432.45
Median TTFT (ms): 297.00
P99 TTFT (ms): 16566.46
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 40.01
Median TPOT (ms): 39.77
P99 TPOT (ms): 82.68
---------------Inter-token Latency----------------
Mean ITL (ms): 40.01
Median ITL (ms): 23.60
P99 ITL (ms): 327.76
----------------End-to-end Latency----------------
Mean E2EL (ms): 13634.15
Median E2EL (ms): 10449.67
P99 E2EL (ms): 30510.40

without kvcahed TTFT faster.
Is there a problem with my kvcached startup script?
Or are there other reasons? This has been bothering me for several days. Please advise, thank you very much.

[cui36]

@yugenlgy Thanks for using our system! I will take a look these days.