Skip to content

Add VLLM_PROFILE_* flags to V1#1203

Merged
kzawora-intel merged 10 commits intohabana_mainfrom
dev/madamczyk/v1_add_profiling
May 8, 2025
Merged

Add VLLM_PROFILE_* flags to V1#1203
kzawora-intel merged 10 commits intohabana_mainfrom
dev/madamczyk/v1_add_profiling

Conversation

@madamczyk-intel
Copy link
Copy Markdown

@madamczyk-intel madamczyk-intel commented May 6, 2025 •
edited by github-actions Bot
Loading

Introduces two new env variables VLLM_PROFILE_PROMPT and VLLM_PROFILE_DECODE. When either is specified, vllm will profile a single batch with given parameters.

VLLM_PROFILE_PROMPT=<BS>,<QUERY_LEN>,<MAX_BLOCKS>
VLLM_PROFILE_DECODE=<BS>,<TOTAL_BLOCKS>

BS - batch size
QUERY_LEN - query length
MAX_BLOCKS - max context blocks used in a batch
TOTAL_BLOCKS - total number of blocks used in a batch

When both variables are present a single mixed batch (profile + decode) will be created.

@madamczyk-intel
Add initial support for profiling flags
51928e9 
Signed-off-by: Michal Adamczyk <madamczyk@habana.ai>
@madamczyk-intel
Merge remote-tracking branch 'origin/habana_main' into dev/madamczyk/…
9fed5cc 
…v1_add_profiling
@madamczyk-intel
Cleanup
28c55c4 
Signed-off-by: Michal Adamczyk <madamczyk@habana.ai>
@madamczyk-intel
Cleanup
101dd4a 
Signed-off-by: Michal Adamczyk <madamczyk@habana.ai>
@madamczyk-intel
Add warning when there is mismatch between use_graphs and enforce_eager
5f75fa1 
Signed-off-by: Michal Adamczyk <madamczyk@habana.ai>
@madamczyk-intel
Copy link
Copy Markdown
Author

madamczyk-intel commented May 6, 2025

/run-gaudi-tests

@michalkuligowski
Copy link
Copy Markdown

michalkuligowski commented May 6, 2025

/run-gaudi-tests

@madamczyk-intel
Copy link
Copy Markdown
Author

madamczyk-intel commented May 7, 2025

/run-gaudi-tests

@madamczyk-intel
Copy link
Copy Markdown
Author

madamczyk-intel commented May 7, 2025

/run-gaudi-tests

@madamczyk-intel
Switch prompt configuration to (bs, query_len, num_blocks)
bbbafd9 
Signed-off-by: Michal Adamczyk <madamczyk@habana.ai>
@madamczyk-intel
Copy link
Copy Markdown
Author

madamczyk-intel commented May 7, 2025

/run-gaudi-tests

@kzawora-intel kzawora-intel merged commit 6c3e443 into habana_main May 8, 2025
43 checks passed
@kzawora-intel kzawora-intel deleted the dev/madamczyk/v1_add_profiling branch May 8, 2025 12:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kzawora-intel kzawora-intel kzawora-intel approved these changes

@michalkuligowski michalkuligowski Awaiting requested review from michalkuligowski michalkuligowski is a code owner

@mgawarkiewicz-intel mgawarkiewicz-intel Awaiting requested review from mgawarkiewicz-intel mgawarkiewicz-intel is a code owner

@vivekgoe vivekgoe Awaiting requested review from vivekgoe

@afierka-intel afierka-intel Awaiting requested review from afierka-intel afierka-intel is a code owner

@xuechendi xuechendi Awaiting requested review from xuechendi xuechendi is a code owner

@jikunshang jikunshang Awaiting requested review from jikunshang jikunshang is a code owner

@mswiniarsk mswiniarsk Awaiting requested review from mswiniarsk mswiniarsk is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@madamczyk-intel @michalkuligowski @kzawora-intel