fix: Fix AMD CI failures with HIP layernorm and PyPI connectivity #13814
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sunxxuns
requested review from
Fridge003,
Kangyan-Zhou,
ispobock and
merrymercy
as code owners
Contributor
|
Note Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported. |
github-actions
bot
added
amd
documentation
sunxxuns
force-pushed
the
ci/add-runner-fallback-support
branch
from
a75f571 to
c984b12
Compare
sunxxuns
requested review from
BBuf,
Edwardf0t1,
HaiShaw,
Ying1123 and
ch-wan
as code owners
sunxxuns
changed the title
sunxxuns
force-pushed
the
ci/add-runner-fallback-support
branch
2 times, most recently
from
0bcc7dc to
7e4f8ca
Compare
sunxxuns
changed the title
sunxxuns
force-pushed
the
ci/add-runner-fallback-support
branch
3 times, most recently
from
56c5f51 to
e73f469
Compare
sunxxuns
requested review from
AniZpZ,
CatherineSue,
FlamingoPg,
hnyls2002,
key4ng,
mickqian,
slin1237,
xiezhq-hermann and
zhyncs
as code owners
github-actions
bot
added
quant
sunxxuns
force-pushed
the
ci/add-runner-fallback-support
branch
2 times, most recently
from
e9eae15 to
0d62bf6
Compare
HaiShaw
approved these changes
Nov 27, 2025
added 2 commits
November 26, 2025 21:42
Two critical AMD CI fixes: 1. Fix fused_add_rms_norm signature in forward_hip (layernorm.py) - Corrected call to match 6-param signature (out, input, residual_out, residual, weight, epsilon) - Fixes TypeError: missing 2 required positional arguments 2. Add retry logic and PyPI mirror fallback (amd_ci_install_dependency.sh) - Retry pip installs up to 3 times with 5s delays - Falls back to Aliyun PyPI mirror on 2nd attempt - Clear Python bytecode cache before installs - Fixes intermittent PyPI connectivity errors All AMD CI tests now pass consistently.
The chown command can fail when multiple CI jobs access the shared pip cache simultaneously. Redirect stderr and continue on errors to prevent flaky failures from race conditions on temporary cache files.
sunxxuns
force-pushed
the
ci/add-runner-fallback-support
branch
from
0d62bf6 to
3d34547
Compare
|
Hi @sunxxuns, When i try this with sglang + vllm (recompiled 0.11.1 but it's true for other vllm versions too as far as i can check) , I get the opposite message : Thanks |
harvenstar
pushed a commit
to harvenstar/sglang
that referenced
this pull request
Dec 4, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR fixes two critical issues causing AMD CI failures:
1. HIP layernorm implementation bug
TypeError: fused_add_rms_norm() missing 2 required positional arguments(out, input, residual_out, residual, weight, epsilon)2. Intermittent PyPI connectivity failures
ERROR: No matching distribution found for black>=24.1.0Test Results
All AMD CI tests now pass consistently ✅
unit-test-backend-*-gpu-amdjobs: PASSaccuracy-test-*-gpu-amdjobs: PASSperformance-test-*-gpu-amdjobs: PASSstage-a-test-*-amdjobs: PASSRun: https://github.com/sgl-project/sglang/actions/runs/19693108888