Skip to content

[bugfix] handle large bucket minimums correctly#235

Merged
kzawora-intel merged 5 commits intohabana_mainfrom
private/kzawora/bucket_min
Sep 4, 2024
Merged

[bugfix] handle large bucket minimums correctly#235
kzawora-intel merged 5 commits intohabana_mainfrom
private/kzawora/bucket_min

Conversation

@kzawora-intel
Copy link
Copy Markdown

This bugfix addresses incorrect lower boundary handling for bucketing

Previous behavior:

INFO 09-03 19:36:28 habana_model_runner.py:564] Prompt bucket config (min, step, max_warmup) bs:[64, 32, 64], seq:[768, 128, 768]
INFO 09-03 19:36:28 habana_model_runner.py:577] Generated 12 prompt buckets: [(32, 128), (32, 256), (32, 384), (32, 512), (32, 640), (32, 768), (64, 128), (64, 256), (64, 384), (64, 512), (64, 640), (64, 768)]
INFO 09-03 19:36:28 habana_model_runner.py:582] Omitted 0 prompt buckets due to exceeded token budget (max_num_batched_tokens=131072)
INFO 09-03 19:36:28 habana_model_runner.py:590] Decode bucket config (min, step, max_warmup) bs:[64, 128, 64], seq:[768, 128, 1024]
INFO 09-03 19:36:28 habana_model_runner.py:601] Generated 8 decode buckets: [(64, 128), (64, 256), (64, 384), (64, 512), (64, 640), (64, 768), (64, 896), (64, 1024)]
INFO 09-03 19:36:28 habana_model_runner.py:606] Omitted 0 decode buckets due to exceeded token budget (max_num_batched_tokens=131072)

Min seq len dimension is set to 768, but buckets with seq_len=128-768 are present

Current behavior:

INFO 09-03 19:45:42 habana_model_runner.py:563] Prompt bucket config (min, step, max_warmup) bs:[64, 32, 64], seq:[768, 128, 768]
INFO 09-03 19:45:42 habana_model_runner.py:576] Generated 1 prompt buckets: [(64, 768)]
INFO 09-03 19:45:42 habana_model_runner.py:581] Omitted 0 prompt buckets due to exceeded token budget (max_num_batched_tokens=131072)
INFO 09-03 19:45:42 habana_model_runner.py:589] Decode bucket config (min, step, max_warmup) bs:[64, 128, 64], seq:[768, 128, 1024]
INFO 09-03 19:45:42 habana_model_runner.py:600] Generated 3 decode buckets: [(64, 768), (64, 896), (64, 1024)]
INFO 09-03 19:45:42 habana_model_runner.py:605] Omitted 0 decode buckets due to exceeded token budget (max_num_batched_tokens=131072)

No bucket with seq_len < 768 is captured

Copy link
Copy Markdown

@madamczyk-intel madamczyk-intel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@kzawora-intel kzawora-intel merged commit a4e1d52 into habana_main Sep 4, 2024
@kzawora-intel kzawora-intel added the habana Issues or PRs submitted by Habana Labs label Sep 5, 2024
@kzawora-intel kzawora-intel deleted the private/kzawora/bucket_min branch October 7, 2024 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

habana Issues or PRs submitted by Habana Labs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants