[Serve] Add default autoscaling params for custom policies

[vaishdho1]

Description

Currently, custom autoscaling policies bypass all standard autoscaling configuration parameters - they must manually implement delay logic, scaling factors, and bounds checking themselves. This PR adds an apply_autoscaling_config decorator that enables custom autoscaling policies to automatically benefit from Ray Serve's standard autoscaling parameters that are embedded in the default policy (upscale_delay_s, downscale_delay_s, downscale_to_zero_delay_s, upscaling_factor, downscaling_factor, min_replicas, max_replicas).

Related issues

Fixes #58622

Implementation Details

• Core implementation (python/ray/serve/autoscaling_policy.py):
  • Added apply_autoscaling_config decorator
  • Refactored delay logic into _apply_delay_logic() helper function
  • Added scaling factor logic for custom policies_apply_scaling_factors() helper function
  • Refactored bounds checking into _apply_bounds() helper function
  • Updated replica_queue_length_autoscaling_policy to use _apply_delay_logic function
• Tests (python/ray/serve/tests/test_autoscaling_policy.py and python/ray/tests/unit/test_autoscaling_policy.py)
  • End-to-end tests verifying delay enforcement for decorated custom policies
  • Tests for scaling factor moderation (upscaling and downscaling)
  • Unit tests for checking each helper function

Added documentation for usage with example

[gemini-code-assist]

Code Review

This pull request introduces a very useful @apply_autoscaling_config decorator to simplify custom autoscaling policies by applying standard configuration parameters automatically. The implementation is well-structured, refactoring existing logic into reusable helper functions. The addition of comprehensive unit and end-to-end tests ensures the new functionality is robust and covers various scenarios, including delays, scaling factors, and edge cases like scaling to zero. The documentation is also updated clearly.

My review includes a critical suggestion to remove leftover development notes from the code and improve handling of a reserved key in the policy state. I've also included a few medium-severity suggestions to improve documentation clarity and test coverage.

[github-actions]

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

[abrarsheikh]

@vaishdho1 is this now blocked?

[vaishdho1]

Yes this is unblocked. I am writing a few tests. Will update this PR today.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[cursor]

Division by zero when policy called with zero replicas

Medium Severity

The _core_replica_queue_length_policy function performs ctx.total_num_requests / target_num_requests where target_num_requests equals config.get_target_ongoing_requests() * num_running_replicas. When current_num_replicas is 0, this causes a ZeroDivisionError. The public function replica_queue_length_autoscaling_policy and default_autoscaling_policy directly call this core function without protection. While internal usage is protected by the apply_autoscaling_config decorator's cold-start path, direct calls to these exported functions will crash.

Additional Locations (1)

• python/ray/serve/autoscaling_policy.py#L278-L291

 

[vaishdho1]

We will not be calling this function directly. In the current flow it is always passed through apply_autoscaling_config decorator.
In tests where this policy is explicitly tested, it is passed through the same decorator.
Should we explicitly mention this or have a check to prevent access to this function directly?

[abrarsheikh]

if its a private function, then prefix it with _

[abrarsheikh]

but its a good idea to guard this function from division by zero without assuming anything about how it get's called.

[abrarsheikh]

should we return 0 when target_num_requests == 0 ?

[vaishdho1]

Fixed this. We should guard num_running_replicas from becoming zero

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[abrarsheikh]

CI failing with

[2026-01-27T07:13:04Z] !!! No API stability annotation found for:
--
[2026-01-27T07:13:04Z] ray.serve.autoscaling_policy.apply_app_level_autoscaling_config
[2026-01-27T07:13:04Z] ray.serve.autoscaling_policy.apply_autoscaling_config
[2026-01-27T07:13:04Z] ray.serve.autoscaling_policy.apply_default_params
[2026-01-27T07:13:04Z] ray.serve.autoscaling_policy.replica_queue_length_autoscaling_policy

except for replica_queue_length_autoscaling_policy all others should be private, i think.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

[cursor]

App-level policy state discarded during cold start

Medium Severity

In _apply_app_level_autoscaling_config, when a deployment is cold-starting, the wrapper uses state_per_deployment[dep_id] (the original state) instead of merging with updated_custom_policy_state.get(dep_id, {}). The user's policy IS called first (line 240), and may return state updates for cold-starting deployments, but those updates are silently discarded. This differs from single-deployment behavior where the policy isn't called during cold start, so there's no state to lose. Stateful app-level policies tracking counters or other state during cold start will experience unexpected state loss.

 

[vaishdho1]

So here, unlike a single deployment case some deployments can have 0 replicas with num_requests>0 and some may not.
Currently, I pass everything through the policy, overwrite those deployments which have 0 replicase and num_requests>0 with cold start path. In this process we will loose user state of those overwritten deployments.
The other alternative is to first apply cold start path to those deployments which start with 0 replicas and num_requests>0 and send the other ones through the policy, this will exactly mimic the deployment level behavior.

[abrarsheikh]

i am not quite following this, but I think its okay to take it as a follow up.

[abrarsheikh]

some nits

[abrarsheikh]

i am not quite following this, but I think its okay to take it as a follow up.

[abrarsheikh]

should we return 0 when target_num_requests == 0 ?

[vaishdho1]

Yes, the original default policy has the same functionality with target_num_requests=0.
For the application autoscaling case, I will keep the cold_start path implementation unchanged. I can create a follow up to look into this separately.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[vaishdho1]

Modified cold_start_path for app level autoscaling to preserve user state. Will pick this up as a follow up for further enhancement.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

[cursor]

Duplicated internal policy state extraction logic

Low Severity

The code block that extracts the internal policy state (creating an internal_policy_state dict with SERVE_AUTOSCALING_DECISION_COUNTERS_KEY) is duplicated verbatim in both _apply_default_params_and_merge_state and _merge_user_state_with_internal_state. This extraction logic could be refactored into a small helper function to eliminate the duplication and reduce maintenance burden.

Additional Locations (1)

• python/ray/serve/autoscaling_policy.py#L173-L179

 

[abrarsheikh]

same here, let's address this as follow up

[cursor]

Bounds applied twice creating redundant computation

Low Severity

Bounds checking is applied twice in the autoscaling flow: first via _apply_bounds inside the _apply_default_params function (called by the decorator), and then again via self.apply_bounds in get_decision_num_replicas. While applying bounds twice is idempotent and doesn't cause incorrect behavior, it creates redundant computation and makes the code flow confusing to understand.

Additional Locations (1)

• python/ray/serve/_private/autoscaling_state.py#L330-L331

 

[abrarsheikh]

let's ticket this and address as a follow up

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

[cursor]

Shared state dict mutation corrupts delay counters

Medium Severity

In _apply_app_level_autoscaling_config, when processing deployments, the functions _merge_user_state_with_internal_state and _apply_default_params_and_merge_state mutate the user_policy_state dict in place via .update(). If an app-level policy returns the same dict object for multiple deployments' state (e.g., {d1: shared_dict, d2: shared_dict}), each deployment's internal delay counter (SERVE_AUTOSCALING_DECISION_COUNTERS_KEY) overwrites the previous one. This causes the delay logic to malfunction because earlier deployments lose their counter state.

Additional Locations (1)

• python/ray/serve/autoscaling_policy.py#L159-L163

 

[vaishdho1]

This looks like a potential issue.
#58857 (comment)
This logic was finalized as part of this discussion where we decided on directly updating the user state.

But if the user actually uses the same state for multiple deployments in application level policies, the delay counters can get corrupted. I think we need to either create a copy of the user state or return a new dict as the merged state.

[abrarsheikh]

let's take that up in a follow up PR. Thanks for staying on top @vaishdho1