Higher Performance with Lower SM Occupancy through Zero-Copy and TMA Offloading#453
Open
monethuang1 wants to merge 26 commits intodeepseek-ai:mainfrom
Open
Higher Performance with Lower SM Occupancy through Zero-Copy and TMA Offloading#453monethuang1 wants to merge 26 commits intodeepseek-ai:mainfrom
monethuang1 wants to merge 26 commits intodeepseek-ai:mainfrom
Conversation
Co-authored-by: Xingyi Li <jerryxyli@tencent.com> Co-authored-by: Xiaojie Huang <monethuang@tencent.com>
Internal commit ac9360c465a4074dd913b885e394b43e1135d986.
Based on internal commit 6943948bd2474b3f36e03de6d1cfed839f199831.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The original Internode Normal Kernel suffers from high GPU SM utilization and underutilized interconnect bandwidth, which constrains prefill performance.
In our optimized version, we apply buffer fusion and TMA offloading to enable true zero-copy communication and maximize NVLink bandwidth usage.
Evaluation on H20 clusters shows significant gains:
Additionally, SM occupancy was reduced by up to 66.7%. The optimized kernel uses only 12 SMs for EP=16 and 8 SMs for EP=32, compared to 24 SMs in the original version.