Can the stop tokens be retained?

[CSWellesSun]

I set the stop token as ., but the output text will not include the stop token, so can the stop tokens be retained in the output?

request:

curl http://localhost:8081/generate \
  -H "Content-Type: application/json" \
  -d '{
    "text": "Once upon a time,",
    "sampling_params": {
      "max_new_tokens": 1024,
      "temperature": 0,
      "stop": ".",
    }
  }'

response:

{"text":" and in a land far, far away, there was a young girl named Lily","meta_info":{"prompt_tokens":6,"completion_tokens":18,"id":"fe8e78ca46ab4f4bb69842202201288c"}}

[merrymercy]

Can you add it by yourself in your script? Or do you have multiple stop strings?

[merrymercy]

The code for handling the stop str is here

sglang/python/sglang/srt/managers/detokenizer_manager.py

Lines 50 to 52 in 9e037c8

	pos = output_strs[i].find(recv_obj.hit_stop_str[i])
	if pos != -1:
	output_strs[i] = output_strs[i][:pos]

You can try to hardcode for your case or you can contribute a PR to add an option.

[merrymercy]

@CSWellesSun You are welcome to try this new JSON decode feature on the main branch. We landed some additional optimizations to make it faster https://github.com/sgl-project/sglang/tree/main?tab=readme-ov-file#json-decoding

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.