[router] refactor generate to use new pipeline arch

[slin1237]

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Summary of Changes

Hello @slin1237, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

The pull request refactors the generate endpoint to use a new pipeline architecture, standardizing its response format and consolidating request processing logic. This involves introducing new GenerateResponse structs, migrating request handling from individual routers to a centralized pipeline, and implementing comprehensive streaming and non-streaming response processing within the pipeline, including token decoding, stop sequence handling, and logprob conversion. Utility functions for logprob conversion and finish reason parsing were also added.

Highlights

• Intent: This pull request refactors the generate endpoint to fully utilize the new pipeline architecture, aiming to consolidate request processing logic and standardize the generate API response format. This brings the generate endpoint in line with how chat requests are handled, improving modularity and maintainability.
• Key Changes: 1. New Generate Response Structures: Introduced GenerateResponse, GenerateMetaInfo, and GenerateFinishReason structs in sgl-router/src/protocols/spec.rs to define a standardized and richer response format for the generate API, including detailed metadata and logprobs.

2. Pipeline Integration: The route_generate_impl methods in both sgl-router/src/routers/grpc/router.rs and sgl-router/src/routers/grpc/pd_router.rs have been significantly simplified. All generate request processing, including worker selection, request building, and stream handling (both streaming and non-streaming), is now delegated to self.pipeline.execute_generate().
3. Consolidated Logic: Removed extensive, duplicated logic for generate request resolution, worker selection, bootstrap metadata injection, and direct stream handling from pd_router.rs and router.rs.
4. Pipeline Response Processing: The process_generate_response method in sgl-router/src/routers/grpc/pipeline.rs has been fully implemented. It now handles collecting responses from workers, merging prefill logprobs (for dual-worker setups), processing tokens through a stop decoder, and constructing the new GenerateResponse objects.
5. Streaming Generate Handler: A new process_streaming_generate method has been added to sgl-router/src/routers/grpc/streaming.rs. This method manages the Server-Sent Events (SSE) stream for generate requests, handling chunking, text accumulation, token counting, and logprob inclusion for both single and dual-worker (Prefill-Decode) execution modes.
6. Utility Functions: Added helper functions in sgl-router/src/routers/grpc/utils.rs for converting protobuf logprob formats to the new Generate format and for robustly parsing finish_reason strings into the new GenerateFinishReason enum.

• Impact: This refactoring significantly improves the architecture of the router by centralizing generate request handling within the pipeline. It reduces code duplication, enhances consistency across different request types, and provides a more structured and informative response format for the generate API. The changes streamline future development and maintenance of the router's core functionalities.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request is a great step forward in refactoring the generate endpoint to use the new pipeline architecture. Centralizing the logic for both regular and prefill-decode (PD) modes into the pipeline significantly improves code clarity and maintainability by removing a lot of duplicated code from the router implementations.

I've identified a few areas for improvement in the new implementation details, mainly concerning the robustness of response parsing and the accuracy of some metadata fields. Please see my detailed comments below.