[router] harmony responses api streaming support

[slin1237]

Overview

This PR implements complete streaming support for Harmony-enabled models in the Responses API, including incremental token parsing, 14 SSE event types, channel-based response processing, and MCP tool loop orchestration.

Scope:

• Responses API streaming with 14 SSE event types
• Single and Dual (PD) stream execution modes
• MCP tool loop integration with multi-iteration support
• Incremental token-by-token parsing via HarmonyParserAdapter
• Channel-based output routing (analysis/commentary/final)

---

Data Flow

gRPC Workers (SGLang)
  ↓ (token_ids stream)
AbortOnDropStream
  ↓ (chunk by chunk)
Streaming Processor
  ↓ (per token)
HarmonyParserAdapter.parse_chunk()
  ↓ (channel deltas)
Channel Routing
  ├─ Analysis → Reasoning output item (wrapper events only)
  ├─ Commentary → Tool calls (accumulated for detection)
  └─ Final → Message output item (text deltas streamed)
  ↓
ResponseStreamEventEmitter
  ↓ (14 SSE event types)
HTTP/2 SSE Stream
  ↓
Client (receives events in real-time)

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Summary of Changes

Hello @slin1237, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the Harmony Responses API by adding comprehensive streaming support. This allows clients to receive real-time updates on the progress of a response, including intermediate tool calls, analysis, and partial text generation, via Server-Sent Events (SSE). The changes involve a new streaming endpoint, modifications to the core processing pipeline to handle streaming requests, and a robust event emission system to provide a dynamic and interactive user experience.

Highlights

• Streaming Support for Harmony Responses API: Introduced a new serve_harmony_responses_stream function to provide real-time streaming capabilities for the Harmony Responses API, utilizing Server-Sent Events (SSE).
• SSE Event Emission: Implemented detailed SSE event emission for various stages of the response lifecycle, including response.created, response.in_progress, mcp_list_tools, output_item.added, text.delta, mcp_call.in_progress, and response.completed.
• Pipeline Integration: Modified the Harmony response processing pipeline to differentiate between streaming and non-streaming requests, allowing streaming requests to return raw execution results for token-level processing.
• Modular Streaming Processor: Created dedicated streaming processor functions (process_responses_iteration_stream, process_responses_single_stream, process_responses_dual_stream) to manage the parsing of gRPC stream chunks and the emission of corresponding SSE events.
• Public Visibility for Streaming Utilities: Adjusted the visibility of ResponseStreamEventEmitter and its associated types and methods to pub to support external usage within the streaming implementation.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces streaming support for the Harmony responses API, a significant feature enhancement. The implementation involves substantial changes, including a new streaming service function, modifications to the request pipeline to accommodate streaming, and a new streaming processor. The overall approach is sound, but I have identified two areas for improvement. One is a potential panic in error handling that could crash the streaming task, and the other is significant code duplication in the new streaming processor, which impacts maintainability. Addressing these points will improve the robustness and quality of the new streaming functionality.

[slin1237]