fix(llm): add stop_sequences parity for tool completions

[G7CNF]

Summary

This closes the tool/completion parameter parity gap for stop_sequences.

What changed

• Added stop_sequences: Option<Vec<String>> to ToolCompletionRequest.
• Added with_stop_sequences(...) builder on ToolCompletionRequest.
• Updated strip_unsupported_tool_params() to strip stop_sequences just like completion requests.
• Threaded stop_sequences through worker/orchestrator proxy DTOs for tool completions.
• Passed OpenAI-compatible stop values into tool completion requests in both streaming and non-streaming HTTP paths.
• Wired provider usage where supported:
  • bedrock: uses tool-request stop sequences in inference config.
  • nearai_chat: serializes stop for both completion and tool-completion requests.

Tests

• Added regression test in provider tests:
  • test_strip_unsupported_tool_params_strips_stop_sequences
• Updated existing request-serialization/tool-cache test fixtures for new field.

Closes #872

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the LLM system by introducing the stop_sequences parameter for tool completion requests, ensuring consistent behavior and capabilities across different types of LLM interactions. This change involves updating core request structures, API handlers, and provider-specific implementations to correctly process and propagate stop sequences, thereby improving control over tool-generated responses.

Highlights

• Tool Completion Parameter Parity: The stop_sequences parameter has been added to ToolCompletionRequest to achieve parity with regular completion requests, closing a previous gap in functionality.
• API Integration: The stop_sequences parameter is now threaded through the worker/orchestrator proxy DTOs and passed into tool completion requests for both streaming and non-streaming HTTP paths.
• Provider Support: Specific LLM providers like Bedrock and NearAI Chat have been updated to utilize the stop_sequences parameter in their respective inference configurations and request serialization.
• Parameter Stripping Logic: The strip_unsupported_tool_params() function has been updated to correctly strip stop_sequences for providers that do not support it, similar to how other parameters are handled.
• Testing: A new regression test was added to verify the stripping of unsupported stop_sequences for tool requests, and existing test fixtures were updated to reflect the new field.

Changelog

• src/channels/web/openai_compat.rs
  • Integrated stop_sequences from the incoming request into ToolCompletionRequest for both chat_completions_handler and handle_streaming functions.
• src/llm/bedrock.rs
  • Modified the build_inference_config call to pass request.stop_sequences to the Bedrock provider.
• src/llm/nearai_chat.rs
  • Added stop: req.stop_sequences to the ChatCompletionRequest for both standard and tool-enabled chat completions.
  • Introduced a stop: Option<Vec<String>> field to the ChatCompletionRequest struct.
  • Updated test fixtures to include stop: None in ChatCompletionRequest instances.
• src/llm/provider.rs
  • Added pub stop_sequences: Option<Vec<String>> to the ToolCompletionRequest struct.
  • Initialized stop_sequences to None in ToolCompletionRequest::new.
  • Implemented a with_stop_sequences builder method for ToolCompletionRequest.
  • Removed an outdated comment regarding StopSequences applicability to tool calls.
  • Updated strip_unsupported_tool_params to clear req.stop_sequences if the parameter is unsupported.
  • Added a new test test_strip_unsupported_tool_params_strips_stop_sequences to validate the stripping logic.
• src/llm/response_cache.rs
  • Updated ToolCompletionRequest initialization in test_cache_key_includes_tool_choice to include stop_sequences: None.
• src/orchestrator/api.rs
  • Mapped req.stop_sequences from the incoming request to the stop_sequences field of the ToolCompletionRequest.
• src/worker/api.rs
  • Added pub stop_sequences: Option<Vec<String>> to the ProxyToolCompletionRequest struct.
  • Cloned request.stop_sequences when constructing ProxyToolCompletionRequest in WorkerHttpClient::llm_complete_with_tools.

Activity

• Added a new regression test test_strip_unsupported_tool_params_strips_stop_sequences to src/llm/provider.rs.
• Updated existing request-serialization and tool-cache test fixtures in src/llm/nearai_chat.rs and src/llm/response_cache.rs to accommodate the new stop_sequences field.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly adds stop_sequences support for tool completions, achieving parity with regular completions. The changes are well-implemented across the different layers, from the OpenAI compatibility layer through the orchestrator/worker DTOs and down to the Bedrock and NearAI providers. A regression test has also been added. My only suggestion is to refactor some duplicated code in src/channels/web/openai_compat.rs to improve maintainability, which I've detailed in a specific comment.

[zmanian]

Review: Add stop_sequences to ToolCompletionRequest for parity

Clean, well-scoped fix. ToolCompletionRequest was silently dropping stop sequences from OpenAI compat requests. Now wired through all providers.

Positives:

• stop_sequences: Option<Vec<String>> added to ToolCompletionRequest with builder method
• All providers updated: bedrock, nearai_chat, openai_compat, proxy (orchestrator + worker)
• Refactored chat_completions_handler into build_completion_request/build_tool_request helpers, eliminating 3x code duplication between streaming/non-streaming/tool paths
• strip_unsupported_tool_params correctly handles StopSequences for providers that don't support it
• Test for strip behavior

LGTM.