[router][grpc] Fix request_id extraction when n > 1

[CatherineSue]

Motivation

When n>1, grpc_request_manager.py relies on rid_to_index to map the rid to its index.

However, the current rid extraction is wrong. As in _handle_batch_output, rid is stored in output_data["request_id"]

sglang/python/sglang/srt/entrypoints/grpc_request_manager.py

Lines 564 to 566 in 4ed67c2

	output_data = {
	"request_id": rid,
	"token_ids": batch_out.output_ids[i] if batch_out.output_ids else [],

Modifications

• Fix rid extraction

Accuracy Tests

Before

request.n = 2

No index = 1

After

There is index = 1

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Summary of Changes

Hello @CatherineSue, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical bug in the gRPC request manager where the request_id was not being correctly extracted from the response when handling multiple requests (i.e., n > 1). This issue prevented the proper mapping of request IDs to their corresponding indices for client-side ordering. The change ensures that the correct request_id is identified, thereby restoring the intended functionality for response indexing.

Highlights

• Request ID Extraction Fix: Corrected the logic for extracting the request_id in grpc_request_manager.py. Previously, when n > 1, the system incorrectly attempted to retrieve the ID from response["meta_info"]["id"], leading to improper mapping. The fix now directly accesses response["request_id"].

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request addresses a bug in grpc_request_manager.py where the request ID was being extracted incorrectly for streaming responses when n > 1. The change correctly modifies the logic to look for request_id at the top level of the response dictionary, instead of a non-existent id field within meta_info. The fix is accurate and improves the code's correctness and clarity. The change is well-contained and I have no further suggestions.