[model-gateway] /parse/easoning and parse/function_call for sgl-model-gateway

[UbeCc]

Motivation

Now, sglang server has separate_reasoning and parse_function_call apis for content extraction. But when using sgl-model-gateway, to extract fn call, we should first list_workers and then call a worker, which will interfere the decoding process. So we can implement the methods in the gateway side for higher concurrency.

BTW, now sglang itself holds the apis as separate_reasoning and parse_function_call which can be aligned by parse/ prefix. So after discussing with @slin1237 , I implement the apis as /parse/reasoning and /parse/function_call

Modifications

implement apis in server.rs

• POST /parse/reasoning
• POST /parse/function_call

implement protocals

• ParseFunctionCallRequest
• SeparateReasoningRequest

implement src/routers/parse for handlers

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[UbeCc]

for example,

curl -X POST http://localhost:8000/parse_function_call \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Call get_weather <tool_call>{\"function_name\": \"get_weather\", \"parameters\": {\"location\": \"Singapore\"}}</tool_call>",
    "tool_call_parser": "json",
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Get weather",
          "parameters": {
            "type": "object",
            "properties": {
              "location": {"type": "string"}
            },
            "required": ["location"]
          }
        }
      }
    ]
  }'

curl -X POST http://localhost:8000/separate_reasoning \
  -H "Content-Type: application/json" \
  -d '{
    "text": "<think>Step 1: Understand the problem</think>Final answer",
    "reasoning_parser": "step3"
  }

[slin1237]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces new endpoints /parse/reasoning and /parse/function_call to the model gateway, allowing for parsing of model outputs at the gateway level. The implementation is well-structured, adding new protocol definitions, router handlers, and server routes. I've included two main points of feedback: one to refactor the new handler logic for better readability and maintainability, and a second, more critical point about fixing the new integration tests, which currently don't exercise the parsing logic due to an issue in the test setup.

[gemini-code-assist]

The tests for the new parser endpoints are not effective because the parser factories (tool_parser_factory and reasoning_parser_factory) are not initialized in the AppContext for the test environment. These factories are currently only created in gRPC mode, but the new endpoints are HTTP-based.

As a result, the handlers always return SERVICE_UNAVAILABLE, and the actual parsing logic is never executed. The tests are passing but not validating the intended functionality.

To fix this, you should manually inject the parser factories into the AppContext during the test setup. After doing this, you should also update the test assertions to check for StatusCode::OK and validate the response body to ensure parsing was successful.

For example, in test_parse_function_call_success, you should change the assertion to assert_eq!(resp.status(), StatusCode::OK); and add checks for the content of the JSON response.

        // Create app context and inject parser factories for testing
        let app_context_base = common::create_test_context(config.clone()).await;
        let mut app_context_mut = (*app_context_base).clone();
        app_context_mut.tool_parser_factory =
            Some(sgl_model_gateway::tool_parser::ParserFactory::new());
        app_context_mut.reasoning_parser_factory =
            Some(sgl_model_gateway::reasoning_parser::ParserFactory::new());
        let app_context = Arc::new(app_context_mut);

        // Create router
        let router = RouterFactory::create_router(&app_context).await.unwrap();
        let router = Arc::from(router);

[gemini-code-assist]

The parse_function_call and parse_reasoning handlers contain deeply nested match statements, which can be difficult to read and maintain. This is often referred to as the 'pyramid of doom'.

I suggest refactoring both functions to use early returns (e.g., with let-else) to flatten the control flow. Additionally, creating a helper function for error responses would reduce code duplication and improve clarity.

Here is a suggested refactoring for the entire file:

//! Parser handlers for function calls and reasoning extraction

use std::sync::Arc;

use axum::{
    http::StatusCode,
    response::{IntoResponse, Response},
    Json,
};
use tracing::error;

use crate::{
    app_context::AppContext,
    protocols::parser::{ParseFunctionCallRequest, SeparateReasoningRequest},
};

/// Creates a standardized JSON error response.
fn create_error_response(status: StatusCode, error_msg: String) -> Response {
    (
        status,
        Json(serde_json::json!({
            "error": error_msg,
            "success": false
        })),
    )
        .into_response()
}

/// Parse function calls from model output text
pub async fn parse_function_call(
    context: Option<&Arc<AppContext>>,
    req: &ParseFunctionCallRequest,
) -> Response {
    let Some(ctx) = context else {
        return create_error_response(
            StatusCode::SERVICE_UNAVAILABLE,
            "Context not initialized".to_string(),
        );
    };

    let Some(factory) = &ctx.tool_parser_factory else {
        return create_error_response(
            StatusCode::SERVICE_UNAVAILABLE,
            "Tool parser factory not initialized".to_string(),
        );
    };

    let Some(pooled_parser) = factory.registry().get_pooled_parser(&req.tool_call_parser) else {
        return create_error_response(
            StatusCode::BAD_REQUEST,
            format!("Unknown tool parser: {}", req.tool_call_parser),
        );
    };

    let parser = pooled_parser.lock().await;
    match parser.parse_complete(&req.text).await {
        Ok((remaining_text, tool_calls)) => (
            StatusCode::OK,
            Json(serde_json::json!({
                "remaining_text": remaining_text,
                "tool_calls": tool_calls,
                "success": true
            })),
        )
            .into_response(),
        Err(e) => {
            let error_msg = format!("Failed to parse function calls: {}", e);
            error!("{}", error_msg);
            create_error_response(StatusCode::BAD_REQUEST, error_msg)
        }
    }
}

/// Parse and separate reasoning from normal text
pub async fn parse_reasoning(
    context: Option<&Arc<AppContext>>,
    req: &SeparateReasoningRequest,
) -> Response {
    let Some(ctx) = context else {
        return create_error_response(
            StatusCode::SERVICE_UNAVAILABLE,
            "Context not initialized".to_string(),
        );
    };

    let Some(factory) = &ctx.reasoning_parser_factory else {
        return create_error_response(
            StatusCode::SERVICE_UNAVAILABLE,
            "Reasoning parser factory not initialized".to_string(),
        );
    };

    let Some(pooled_parser) = factory.registry().get_pooled_parser(&req.reasoning_parser) else {
        return create_error_response(
            StatusCode::BAD_REQUEST,
            format!("Unknown reasoning parser: {}", req.reasoning_parser),
        );
    };

    let mut parser = pooled_parser.lock().await;
    match parser.detect_and_parse_reasoning(&req.text) {
        Ok(result) => (
            StatusCode::OK,
            Json(serde_json::json!({
                "normal_text": result.normal_text,
                "reasoning_text": result.reasoning_text,
                "success": true
            })),
        )
            .into_response(),
        Err(e) => {
            let error_msg = format!("Failed to separate reasoning: {}", e);
            error!("{}", error_msg);
            create_error_response(StatusCode::BAD_REQUEST, error_msg)
        }
    }
}