[Serve] [Docs] Async io + Ray Serve best practices

[abrarsheikh]

https://anyscale-ray--58909.com.readthedocs.build/en/58909/serve/advanced-guides/asyncio-best-practices.html

[marwan116]

Thank you so much for putting this guide together, really clarifies how to think of asynchronous execution with Ray Serve.

I left some comments which I am curious about.

[marwan116]

Should we explicitly encourage Ray core usage ? For example remote actors require proper autoscaling and observability tooling to help scale and debug when things go wrong. Can we instead advocate for composition in this case ?

Here is what cursor suggested as an alternative section:

### Using serve composition for true parallelism

With serve composition, you can split CPU-heavy work into separate deployments that run in independent processes:

```python
@serve.deployment
class HeavyCompute:
    def compute(self, x):
        # Heavy compute runs in its own deployment process.
        return x * x


@serve.deployment
class ParallelProcessor:
    def __init__(self, compute_handle):
        self._compute_handle = compute_handle

    async def __call__(self, request):
        values = [1, 2, 3, 4]
        # Call the composed deployment in parallel.
        refs = [self._compute_handle.compute.remote(v) for v in values]
        results = await asyncio.gather(*[ref for ref in refs])
        return {"results": results}


# Compose the deployments.
app = ParallelProcessor.bind(HeavyCompute.bind())

[abrarsheikh]

The cluster should still be able to autoscale even if ray tasks are created from serve application. But good point about observability. Creating another deployment and offloading the work there has the same problem because then that deployment becomes blocking, so we just kick the can forward to a downstream deployment.

I have seen users use this pattern especially in OSS. So I am inclined to keep this but will mark this a advanced usage pattern, call out the observablity pitfall and not mention it repeatedly in the document.

[marwan116]

Sounds good to me about calling it an advanced pattern and leaving it to a dedicated section.

Yes I agree remote tasks should scale just fine. In my mind with composition they can set target and max requests to a low number close to 1 and use serve's queue based replica scaling to parallelize the compute but I recognize this probably adds more performance overhead than directly submitting tasks.

As for my comment on auto scaling - I was referring to remote actors given the document originally stated "use remote tasks or actors" which would require an auto scaling actor pool abstraction...

[angelinalg]

Suggested change

	- You have a `def` method that is currently running on the event loop.
	- You have a `def` method that is currently running on the event loop.

[akshay-anyscale]

if there isn't, can we add telemetry on the usage of this? Worth considering if this should be a deployment level config

[abrarsheikh]

no telemetry on RAY_SERVE_RUN_SYNC_IN_THREADPOOL. I cannot make a strong case for this to be a deployment config, env var is fine IMO. But we should turn this on by default in new release.

[edoakes]

Looks very useful overall. A few minor comments inline.

More macro comment: we are promoting the usage of a lot of different environment variable feature flags here.

1. Are there default changes we can make to simplify the matrix? For example, making RAY_SERVE_RUN_SYNC_IN_THREADPOOL on by default would help reduce the support matrix a lot.
2. Should any of these be promoted to proper public APIs instead of env var FFs?

[abrarsheikh]

• Are there default changes we can make to simplify the matrix? For example, making RAY_SERVE_RUN_SYNC_IN_THREADPOOL on by default would help reduce the support matrix a lot.

In the next ray release will announce that we will make this default.

• Should any of these be promoted to proper public APIs instead of env var FFs?

RAY_SERVE_RUN_SYNC_IN_THREADPOOL and RAY_SERVE_RUN_ROUTER_IN_SEPARATE_LOOP can be made deployment-level configs. I think that makes sense.

[edoakes]

RAY_SERVE_RUN_SYNC_IN_THREADPOOL should just be turned on by default w/o adding a config option IMO (to match fastapi)

[abrarsheikh]

RAY_SERVE_RUN_SYNC_IN_THREADPOOL should just be turned on by default w/o adding a config option IMO (to match fastapi)

you are right, sorry I copy pasted the wrong env var, meant to put RAY_SERVE_RUN_USER_CODE_IN_SEPARATE_THREAD

[edoakes]

lgtm