[Serve] Downstream deployments over-provision when receiving Deployme…#60747
Merged
abrarsheikh merged 1 commit intomasterfrom Feb 4, 2026
Merged
[Serve] Downstream deployments over-provision when receiving Deployme…#60747abrarsheikh merged 1 commit intomasterfrom
abrarsheikh merged 1 commit intomasterfrom
Conversation
…ntResponse arguments from slow upstream Signed-off-by: abrar <abrar@anyscale.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request correctly addresses an over-provisioning issue in downstream deployments by resolving request arguments before they are counted as queued. The logic change in router.py is direct and well-commented, and the new test case in test_autoscaling_policy.py effectively validates the fix. I have one suggestion to make the test even more robust against potential timing issues.
|
|
||
| # Wait for all 5 requests to be blocked at SlowUpstream (waiting on signal) | ||
| wait_for_condition(lambda: ray.get(signal.cur_num_waiters.remote()) == 5) | ||
|
|
Contributor
There was a problem hiding this comment.
To make this test more robust against timing-related flakiness, it would be beneficial to add a short time.sleep() after waiting for the requests to be blocked and before asserting the number of replicas. This ensures that the autoscaler has had sufficient time to make a (potentially incorrect) scaling decision. Given upscale_delay_s is 0.2s, a sleep of 0.5s should be adequate.
Suggested change
| # Give the autoscaler time to potentially make a wrong decision. | |
| # A sleep duration longer than upscale_delay_s (0.2s) ensures that | |
| # we would have seen an upscale event if the fix was not effective. | |
| time.sleep(0.5) | |
akyang-anyscale
approved these changes
Feb 4, 2026
tiennguyentony
pushed a commit
to tiennguyentony/ray
that referenced
this pull request
Feb 7, 2026
ray-project#60747) fixes ray-project#60624 Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: tiennguyentony <46289799+tiennguyentony@users.noreply.github.com>
tiennguyentony
pushed a commit
to tiennguyentony/ray
that referenced
this pull request
Feb 7, 2026
ray-project#60747) fixes ray-project#60624 Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: tiennguyentony <46289799+tiennguyentony@users.noreply.github.com>
tiennguyentony
pushed a commit
to tiennguyentony/ray
that referenced
this pull request
Feb 7, 2026
ray-project#60747) fixes ray-project#60624 Signed-off-by: abrar <abrar@anyscale.com>
Kunchd
pushed a commit
to Kunchd/ray
that referenced
this pull request
Feb 17, 2026
ray-project#60747) fixes ray-project#60624 Signed-off-by: abrar <abrar@anyscale.com>
ans9868
pushed a commit
to ans9868/ray
that referenced
this pull request
Feb 18, 2026
ray-project#60747) fixes ray-project#60624 Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Adel Nour <ans9868@nyu.edu>
Aydin-ab
pushed a commit
to kunling-anyscale/ray
that referenced
this pull request
Feb 20, 2026
ray-project#60747) fixes ray-project#60624 Signed-off-by: abrar <abrar@anyscale.com>
peterxcli
pushed a commit
to peterxcli/ray
that referenced
this pull request
Feb 25, 2026
ray-project#60747) fixes ray-project#60624 Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>
peterxcli
pushed a commit
to peterxcli/ray
that referenced
this pull request
Feb 25, 2026
ray-project#60747) fixes ray-project#60624 Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: peterxcli <peterxcli@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fixes #60624