[Feature] Tool Call Roadmap

[CatherineSue]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Add a list of issues need to resolve in tool call.

Track for Tool Call Issues

High Piority

Issues related to accuracy, consistency, and performance.

• Multiple Tool Call Support for MistralDetector and Qwen25Detector
  feat: Improve Mistral and Qwen25 function call parsing #6597

• JSON Double Dumping Behavior

• ToolCallItem.tool_index not following OpenAI API
  refactor(tool call): Fix BaseFormatDetector tool_index issue and refactor parse_streaming_increment #6715
  fix(deepseekv3): Fix DeepSeekV3Detector tool_index assignment and multi-tool call streaming support #6655
  fix(tool call): Fix tool_index in PythonicDetector and issues with mixed output in non-streaming #6678

---

Medium Priority

Issues that are not immediate, such as features still WIP, or needs refactor, or edge cases.

• Tests for get_structure_tag in FunctionCallParser

• DeepSeekV3Dectector may have issues with Multiple Tool Calls Streaming Parsing
  fix(deepseekv3): Fix DeepSeekV3Detector tool_index assignment and multi-tool call streaming support #6655

Related resources

No response

[CatherineSue]

- Multiple Tool Call Support for MistralDetector and Qwen25Detector

Assignee: @CatherineSue
PR: #6597

• MistralDetector

  • detect_and_parse does not support parallel function calls.
  • Expected format:

    [TOOL_CALLS] [{"name":"func1", "arguments":{...}}, {"name":"func2", "arguments":{...}}]

• Qwen25Detector

  • bot_token and eot_token should include `\n, and they wrap around a single function call.
  • Full format:

    <tool_call>\n{"name":"func1", "arguments":{...}}\n</tool_call>\n<tool_call>\n{"name":"func2", "arguments":{...}}\n</tool_call>
    

  • Implications:
    • BaseFormatDetector.parse_streaming_increment is incompatible with multiple calls in this format.
    • EBNFComposer.build_ebnf needs an updated interface to support models like Qwen25, where each tool call is wrapped individually.

[CatherineSue]

- JSON Double Dumping Behavior

Assignee: @shuaills

PR: #6559

• Problem:
  • Some detectors load a function call string and pass it to BaseFormatDetector.parse_base_json().
  • If the string was already JSON-encoded, this results in a double json.loads(), leading to escaped quotes in the final result like \".

[CatherineSue]

- ToolCallItem.tool_index not following OpenAI API

Problem: DeepseekV3Detector currently assign tool_index based on its index in the tools. As a result, every function call with the same name will be assigned to the same index. PythonicDetector does not return tool_index at all.

Root Cause
The real issue that causes this problem is that we have this tool_call_index assignment in BaseFormatDetector.parse_streaming_increment. But we don't do any checks to enforce this behavior for an inherited class. As a result, it is very easy to miss the implementation in a new detector class.

We should either do this in the adapter or protocol, or find a better way to enforce the implementation if a Detector can't use BaseFormatDetector.

Expected Behavior: tool_index should be the i-th tool calls in the tool_calls list.

sglang/python/sglang/srt/function_call/deepseekv3_detector.py

Lines 86 to 90 in ed0c303

	self._tool_indices = {
	tool.function.name: i
	for i, tool in enumerate(tools)
	if tool.function and tool.function.name
	}

sglang/python/sglang/srt/function_call/deepseekv3_detector.py

Line 106 in ed0c303

tool_index=self._tool_indices.get(func_name, 0),

Examples from openai sdk:
Tow calls to get_weather, different index.

ChatCompletionChunk(id='chatcmpl-BaZSRwx8vTQrjWwt1AZmOPRoxzENv', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=0, id=None, function=ChoiceDeltaToolCallFunction(arguments='nce"}', name=None), type=None)]), finish_reason=None, index=0, logprobs=None)], created=1748055807, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier='default', system_fingerprint='fp_a288987b44', usage=None)
ChatCompletionChunk(id='chatcmpl-BaZSRwx8vTQrjWwt1AZmOPRoxzENv', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, refusal=None, role=None, tool_calls=[ChoiceDeltaToolCall(index=1, id='call_mOHHKqGAleVBRbEjilzyegzP', function=ChoiceDeltaToolCallFunction(arguments='', name='get_weather'), type='function')]), finish_reason=None, index=0, logprobs=None)], created=1748055807, model='gpt-4o-2024-08-06', object='chat.completion.chunk', service_tier='default', system_fingerprint='fp_a288987b44', usage=None)

[CatherineSue]

- Tests for get_structure_tag in FunctionCallParser

NOTE: follow up with the latest Xgrammar backend structural tag support, this may need refactoring.

• Current test coverage is minimal (only function.strict case in test_openai_function_calling.py).
• Current logic assumes:
  • One StructuresResponseFormat per tool call.
  • info.begin acts as the Detector.bot_token.

StructuresResponseFormat(
    begin=info.begin,
    schema=schema,  # type: ignore
    end=info.end,
)

Problem

This assumes the bot_token and eot_token wrap each individual function call, not the whole sequence.

Behavior

For models like Mistral (which expect a single bot_token like [TOOL_CALLS] before all function calls), Llama-3.2-1B-Instruct (sometimes emits <|python_tag|> before all function calls), this may cause:

• Repetitive tool calls with identical function name and arguments.
• Unstable or non-deterministic behavior.

[CatherineSue]

- DeepSeekV3Dectector may have issues with Multiple Tool Calls Streaming Parsing

Problem

It seems DeepSeekV3Detector is not able to parse parallel tool calls correctly in streaming mode.

Behavior

In NON-STREAMING, 2 tool calls returned.
In STREAMING, Content contains the get_tourist_attractions, which is not parsed as tool call chunk.

[CatherineSue]

- Llama32Detector may fail with mixed outputs

Problem

The following output will raise json.JSONDecodeError, despict it contains a valid tool call. And the output would be StreamingParseResult(normal_text="", calls=[])

sglang/python/sglang/srt/function_call/llama32_detector.py

Lines 39 to 69 in c673727

	def detect_and_parse(self, text: str, tools: List[Tool]) -> StreamingParseResult:
	"""Parse function calls from text, handling multiple JSON objects."""
	if "<|python_tag|>" not in text and not text.startswith("{"):
	return StreamingParseResult(normal_text=text, calls=[])
	if "<|python_tag|>" in text:
	normal_text, action_text = text.split("<|python_tag|>")
	else:
	normal_text, action_text = "", text
	# Split by semicolon and process each part
	json_parts = [
	part.strip()
	for part in action_text.split(self.tool_call_separator)
	if part.strip()
	]
	all_actions = []
	for part in json_parts:
	try:
	# Parse each individual JSON object
	action = json.loads(part)
	all_actions.append(action)
	except json.JSONDecodeError as e:
	logger.warning(f"Failed to parse JSON part: {part}")
	logger.warning(f"JSON parse error: {str(e)}")
	continue
	calls = []
	# Only process if we found valid JSON objects
	if all_actions:
	calls = self.parse_base_json(all_actions, tools)
	return StreamingParseResult(normal_text=normal_text, calls=calls)

text = '{"type": "function", "name": "getOpenIncidentsTool", "parameters": {}} \n\nWould you like to know more about a specific open incident?'