Skip to content

feat: use cuda_graph by default for vllm#116

Merged
parthchadha merged 2 commits intomainfrom
pchadha/vllm-cuda-graph
Apr 1, 2025
Merged

feat: use cuda_graph by default for vllm#116
parthchadha merged 2 commits intomainfrom
pchadha/vllm-cuda-graph

Conversation

@parthchadha
Copy link
Copy Markdown
Contributor

What does this PR do ?

Enables cuda graph by default for vllm generation.
For llama-8b I am seeing 10-15% better generation speed compared to eager mode.

Issues

Closes #115.

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this 

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

Signed-off-by: Parth Chadha <pchadha@nvidia.com>
Comment thread nemo_reinforcer/models/generation/vllm.py
@parthchadha parthchadha enabled auto-merge (squash) April 1, 2025 22:56
@parthchadha parthchadha merged commit d9277a8 into main Apr 1, 2025
11 checks passed
@parthchadha parthchadha deleted the pchadha/vllm-cuda-graph branch April 1, 2025 22:58
yfw pushed a commit that referenced this pull request Apr 2, 2025
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
Signed-off-by: Yi-Fu Wu <yifu.wu@gmail.com>
KiddoZhu pushed a commit that referenced this pull request May 6, 2025
Signed-off-by: Parth Chadha <pchadha@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

vllm use cuda graph for generation

2 participants