support for the DeepSeek model by enabling streaming response parsing

[Frank-Jie]

Base on the Pr #5224

Motivation

Enhance the current function call support for the DeepSeek model by enabling streaming response parsing. In scenarios where stream=true, the tool call arguments are received incrementally, aligning with the current tool-call format design(do not response with complete parameters in the final streaming message).

Additionally, it fixes the incompatibility with xgrammar by removing full-width symbols.

Modifications

• Implemented incremental parsing logic for streamed function call chunks in DeepSeekV3 detector.
• Replaced full-width punctuation characters with ASCII equivalents to avoid xgrammar parse errors.
• Updated documentation to reflect the usage of streaming mode.

[zhaochenyang20]

@shuaills @minleminzui could you two take a look?

[Frank-Jie]

Tested with following code:

import requests, json, random, time

test_func_weather = {
   "type": "function",
   "function": {
       "name": "query_weather",
       "description": "Get weather of an location, the user should supply a location first",
       "parameters": {
           "type": "object",
           "properties": {
               "city": {
                   "type": "string",
                   "description": "The city, e.g. Beijing"
               }
           },
           "required": ["city"]
       }
   }
}


test_func_time = {
   "type": "function",
   "function": {
       "name": "get_current_time",
       "description": "Get current local time for a city",
       "parameters": {
           "type": "object",
           "properties": {
               "city": {
                   "type": "string",
                   "description": "The city, e.g. Tokyo"
               }
           },
           "required": ["city"]
       }
   }
}



tools = [test_func_weather, test_func_time]

templates = [

   ("Hows the weather like in {} today", ["Beijing", "Chongqing", "Shanghai", "Xiamen"],"query_weather"),

   ("What time is it in {} now", ["London", "New York", "Tokyo", "Sydney"],"get_current_time")

]

req_base_url = "http://127.0.0.1:30000"
headers = {"Content-Type": "application/json"}
total_failed = 0

for i in range(20):
   tpl, opts, func_name = random.choice(templates)
   param = random.choice(opts)
   user_content = tpl.format(param)

   req_body = {
       "messages": [{"role": "user", "content": user_content}],
       "temperature": 0,
       "max_tokens": 100,
       "model": "deepseek-ai/DeepSeek-V3-0324",
       "tools": tools,
       "stream": True
   }

   print(f"\n--- Test Case {i + 1}: {user_content} ---")
   try:
       res = requests.post(
           req_base_url + "/v1/chat/completions",
           headers=headers,
           json=req_body,
           stream=True
       )
       calls = []
       for line in res.iter_lines(decode_unicode=True):
           if not line or not line.startswith("data:"):
               continue
           data = line[len("data:"):].strip()
           if data == "[DONE]":
               break
           try:
               json_data = json.loads(data)
               body = json.dumps(json_data, indent=2, ensure_ascii=False)
               delta = json_data["choices"][0]["delta"]
               if delta.get("tool_calls"):
                   calls +=delta["tool_calls"]
           except json.JSONDecodeError:
               raise ValueError(f"Invalid JSON response: {data}")
       assert calls[0].get("function").get("name") == func_name

       arg_str = "".join([c.get("function").get("arguments") for c in calls if c.get("function",{}).get("arguments")])
       args_obj = json.loads(arg_str)

       assert list(args_obj.values())[0] in opts
   except Exception as e:
       print(f"[Error] fail: {e}")
       total_failed += 1

   requests.get(req_base_url + "/flush_cache")
   time.sleep(0.5)

print(f"\ntotal fail number：{total_failed}")

it passed

[thyecust]

Very clear code. I can't do a test on this, I can only understand it in my head.

[thyecust]

Suggested change

	begin=">" + name + "\n```json\n",
	end="\n```<",
	trigger=">" + name + "\n```json\n",
	begin=f"> {name} \n```json\n",
	end="\n```<",
	trigger=f"> {name} \n```json\n",

It is clearer. Do you like it?

[thyecust]

Can we add some comments? I understand that this means that we will ignore these tokens self.eot_token, "```", "<｜tool▁call▁end｜>" for now because they are not complete.

[Frank-Jie]

Thanks for comment.
First, i believe both list[ToolCallItem] and List[ToolCallItem] are valid after python3.9
However according to: https://peps.python.org/pep-0585
with content in it:
This PEP proposes to enable support for the generics syntax in all standard collections currently available in the typing module.
list[ToolCallItem] is more recommended after python3.9
while py3.9 can be seen in current yml flie such as:
https://github.com/sgl-project/sglang/blob/main/.github/workflows/release-whl-kernel.yml#L25

[thyecust]

Thank you for your reply, I think the current code is good to me

[minleminzui]

I checked your code using your test code, it works, lgtm

[minleminzui]

@Frank-Jie please don't merge brach 'main' before we ask you to update your branch, thanks

[minleminzui]

@zhaochenyang20 we can merge it

[zhaochenyang20]

I asked lianmin to merge！

[merrymercy]

merged. Thanks everyone