[data][llm] Support configuring HttpRequestUDF resources

[yuchen-ecnu]

Description

This PR introduces resources configuration options in HttpRequestProcessorConfig to provide more flexible resource control for HttpRequestUDF.

Related issues

Closes #60311

Additional information

It has been tested in python/ray/llm/tests/batch/cpu/stages/test_http_request_stage.py.
Users can use this feature with the following code:

config = HttpRequestProcessorConfig(
    url="http://xxxx/v1/chat/completions",
    headers={"Authorization": "Bearer fake_key"},
    qps=2,
    max_retries=3,
    concurrency=(1, 100),
    http_request_stage=HttpRequestStageConfig(
        num_cpus=0.5,
        memory=100000,
    ),
)

[gemini-code-assist]

Code Review

This pull request introduces a resources configuration option in HttpRequestProcessorConfig to allow more flexible resource control for the HttpRequestUDF. The implementation correctly adds the new field and passes the specified resources to the underlying Ray Data map_batches operator. The changes are accompanied by a test case that verifies the new functionality. The code is clear, correct, and a good addition to provide more control over resource allocation.

[jeffreywang-anyscale]

Hey @yuchen-ecnu thanks for your contribution! Could we follow how other stages (e.g. ChatTemplate, Tokenizer) configure their num_cpus (reference) and not introduce the resources attribute? That is, configure the resources at the stage rather than the processor granularity. Please fix DCO as well (git commit -s -m "your commit message" or git commit -s --amend), thank you! cc: @nrghosh

[nrghosh]

Thanks for the detailed issue writeup / PR!

Valid use-case / concern, and we do support fractional cpu support - so this is a good request.

Main gap to address as you point out is that HttpRequestProcessorConfig doesn't expose that for the HTTP stage. For this PR (as Jeff pointed out), would want to align with the existing pattern by adding explicit fields, like

class HttpRequestProcessorConfig(ProcessorConfig):                                                           
    num_cpus: Optional[float] = Field(                                                                       
      default=None,                                                                                        
      description="CPUs per HttpRequestUDF worker. Defaults to 1 if None. "                                
      "For I/O-bound workloads, use fractional values (e.g., 0.1).",                                       
    )                                                                                                        
    memory: Optional[float] = Field(default=None, ...)                                                       
                    

This way we keep the API consistent with other stages + leverage the existing extract_resource_kwargs() utility.

Also

• lint / sign (from Jeff)
• update test to match above pattern ^

[yuchen-ecnu]

Hi @jeffreywang-anyscale @nrghosh , thanks for the detailed comments!

You’re right that we should keep consistency with the existing pattern. I’ve replaced the generic config resource with explicit fields num_cpus and memory, and updated the tests accordingly.
The DCO issue has also been addressed in the latest commit.

Please let me know if there’s anything else I should improve.

[jeffreywang-anyscale]

The direction makes sense, but let's do this at the stage layer (not the processor layer) to be consistent with how other stages configure num_cpus and memory although there is only 1 stage in the HttpRequestProcessor. We can take advantage of

ray/python/ray/llm/_internal/batch/processor/utils.py

Line 44 in d406677

def build_cpu_stage_map_kwargs(

and will not need to invoke extract_resource_kwargs directly. You can follow this example.

[yuchen-ecnu]

Hi @jeffreywang-anyscale , many thanks for the example — it really helped me understand.

I’ve moved the configs to the stage layer using the new HttpRequestStageConfig and added a guard check to prevent users from disabling the only stage in HttpRequestProcessor.

Please let me know if there are any further improvements I should make.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[jeffreywang-anyscale]

Couple of documentation NITs. Thank you again for the contribution, @yuchen-ecnu!

[jeffreywang-anyscale]

typo?

[yuchen-ecnu]

My mistake, updated.

[yuchen-ecnu]

Couple of documentation NITs. Thank you again for the contribution, @yuchen-ecnu!

Hi @jeffreywang-anyscale , many thanks for your kind and patient review. The related docs have been updated.

[jeffreywang-anyscale]

There are some transient failures with Anyscale:

Traceback (most recent call last):
--
File "/workdir/release/ray_release/job_manager/anyscale_job_manager.py", line 85, in _run_job
job_response = self._sdk.create_job(job_request)
File "/usr/local/lib/python3.10/site-packages/anyscale/sdk/anyscale_client/sdk.py", line 552, in create_job
return super().create_job(create_production_job, **kwargs)
File "/usr/local/lib/python3.10/site-packages/anyscale/sdk/anyscale_client/api/default_api.py", line 884, in create_job
return self.create_job_with_http_info(create_production_job, **kwargs)  # noqa: E501
File "/usr/local/lib/python3.10/site-packages/anyscale/sdk/anyscale_client/api/default_api.py", line 963, in create_job_with_http_info
return self.api_client.call_api(
File "/usr/local/lib/python3.10/site-packages/anyscale/sdk/anyscale_client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/usr/local/lib/python3.10/site-packages/anyscale/sdk/anyscale_client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/usr/local/lib/python3.10/site-packages/anyscale/sdk/anyscale_client/api_client.py", line 391, in request
return self.rest_client.POST(url,
File "/usr/local/lib/python3.10/site-packages/anyscale/sdk/anyscale_client/rest.py", line 270, in POST
return self.request("POST", url,
File "/usr/local/lib/python3.10/site-packages/anyscale/sdk/anyscale_client/rest.py", line 229, in request
raise ApiException(http_resp=r)
anyscale_client.exceptions.ApiException: (500)
Reason: Internal Server Error

Retry didn't succeed. Will retry the failed release tests again tomorrow.

out of office

[bveeramani]

Stamp

[xiyuecangxin]

LGTM!