Performance comparison: vLLM with and without kvcached

[Blueblack319]

Hi, I think this looks like a very interesting project.
I'm wondering, however, if vLLM with kvcached shows lower performance than vanilla vLLM due to dynamic memory allocation overhead.
Have you measured the overhead by comparing vanilla vLLM and vLLM with kvcached when running a single model?

[jiarong0907]

Hi @Blueblack319, Yes, we have a benchmark here: https://github.com/ovg-project/kvcached/tree/main/benchmarks/bench_kvcached_overhead.

The overhead will only become siginificantly when the request rate is very high, in which case dynamically sharing GPU might not be a good idea.

[Blueblack319]

I conducted an experiment comparing vLLM performance with and without kvcached (Llama-3.1-8B, A100 80GB).

While I anticipated some overhead, the performance degradation with kvcached was unexpectedly severe. The results show a much larger drop than I had initially supposed. Furthermore, as shown below, I observed significant variance in TTFT across different runs when using kvcached, indicating a major performance inconsistency.

Could you help me understand why the performance is falling so far short of the baseline? I’m wondering if this points to a fundamental issue in my setup.

Command used both systems

• Server: vllm serve /work3/llm_weights/Llama-3.1-8B --disable-log-requests --no-enable-prefix-caching --port=12346 --tensor-parallel-size=1
• Client: vllm bench serve --backend vllm --model /work3/llm_weights/Llama-3.1-8B --endpoint /v1/completions --dataset-name sharegpt --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json --num-prompts 100 --port 12346 --request-rate 5 --save-result --result-filename ./shared_gpu.json

vLLM w/o kvcached

vLLM w/ kvcached

• Trial 1

- Trial 2

[jiarong0907]

This is absolutely abnormal. To help understand the problem, could you try this benchmark? https://github.com/ovg-project/kvcached/tree/main/benchmarks/bench_vmm

[ivanium]

Just saw the issue. Thanks Jiarong for offering the link to the microbenchmark! A bit more context here is that the kvcached's performance sometimes is impacted by the host machine and NVIDIA driver's performance, because it uses the CDUA VMM features extensively. The benchmark Jiarong provided will be helpful for us to understand the concrete performance of the system and GPU VMM performance. Sharing the results of that will be very helpful for performance debugging.

Additionally, it would be helpful to also share the machine (e.g., is it a GCP/AWS instance), CUDA driver version and NVIDIA driver version (shown by nvidia-smi), and vLLM version you are using.

[Blueblack319]

Hi @ivanium, and thanks @jiarong0907 for providing such a useful microbenchmark!

I ran the microbenchmark on two different environments, and the results vary significantly between them. Specifically, I tested this on a server with 4x A100 GPUs and another with 2x H100 GPUs.

While the H100 server successfully reproduced the benchmark results attached in the README.md, the A100 server showed significantly poorer performance. This poor result on the A100 seems consistent with the performance issues previously observed in vLLM when using KV Cache.

Could you help me understand the possible reasons for this performance discrepancy? Additionally, would it be possible to share the environment details (e.g., driver versions, CUDA version, or OS) used for the original microbenchmark?

Detailed logs for both environments are provided below:

A100 server

• bench_vmm.sh

• nvidia-smi

• nvidia-smi topo -m

H100 server

• bench_vmm.sh

• nvidia-smi

• nvidia-smi topo -m

[ivanium]

Thanks for the experiments and glad to hear that you reproduced results with the H100 node!

This performance issue with A100 machine is actually deep and residing in the machine configuration and NVIDIA drivers. Unfortunately, the TLDR is that we'll need fix from NVIDIA side for machines with performance anomaly. Updating CUDA driver might help but there is no guarantee.

A bit more detailed explanation is that this bench_vmm microbenchmark benchmarks the CUDA VMM APIs' performance, and we did observed similar amonalies before on some machines that these APIs are significantly slower as your figures shown. We have reached out to NVIDIA regarding this, and they confirmed this is related to the closed-source internal CUDA driver implementation. However, this does not always happen and we haven't figured out the exact conditions that trigger it. It usually happens with some third-party machines with customized BIOS, host OS kernel, etc. and sometimes related to container cloud. Since only NVIDIA can debug an fix this in the CUDA driver, there is not too much we can do on our side.

[Blueblack319]

Thank you for the detailed response. It clarifies the situation completely.