[Serve] Optimize replica routing request data structures

[abrarsheikh]

1. O(1) Pending Request Lookups

   • Added dict indices (_pending_requests_by_id and _pending_requests_by_model_id) for fast lookups
   • Replaced O(n) linear scans with O(1) dict lookups when finding requests by ID or multiplexed model

2. Cached Replica List

   • Added _replicas_list cache to avoid O(n) dict-to-list conversion on every routing iteration
   • List updated only when replicas change via update_replicas() or on_replica_actor_died()

3. Lazy Cleanup Strategy

   • Done futures are lazily cleaned from _pending_requests_by_model_id during lookups using O(1) popleft()
   • Avoids expensive O(n) removal from deques

4. Optimized Retry Insertion

   • Extracted sorted insertion logic into _insert_pending_request_sorted() helper
   • O(1) fast path for common case (recent retries append to end)

5. Simplified pow_2_router

   • Removed redundant dict creation per routing call
   • Direct lookup via self._replicas[chosen_id] instead of building temporary map

1. random.sample → Direct Selection
2. Lazy Hash Caching (common.py)
3. Metrics Throttling (request_router.py, constants.py)

flamegraph of the router after all the optimization

[gemini-code-assist]

Code Review

This pull request significantly optimizes the replica routing mechanism in Ray Serve by refactoring data structures and lookup logic. The changes introduce dictionary-based indices (_pending_requests_by_id, _pending_requests_by_model_id) for O(1) lookups of pending requests, replacing previous O(N) iterations over deques. Lazy cleanup of completed futures is implemented to prevent memory leaks, and a cached list of replicas (_replicas_list) is maintained to avoid redundant list conversions. These improvements enhance the efficiency of request matching, fulfillment, and replica selection, leading to better performance, especially in high-throughput or multiplexed model scenarios. The code is well-commented, explaining the rationale behind the optimizations.

[harshit-anyscale]

great improvements, nice work!
left some comments, else LGTM