Issue when using choices option. assert temperature <= 1e-5 TypeError: '<=' not supported between instances of 'NoneType' and 'float'

[psych0v0yager]

I ran into the following error when running my code:

Code

import sglang as sgl

sgl.set_default_backend(sgl.RuntimeEndpoint("http://localhost:30000"))

@sgl.function
def test_function(s, test_prompt):
    s += "This is a test, I will provide a prompt and answer to the best of your abilities. Once you are done, end with the word 'END'" + "\n"
    s += "Q: " + test_prompt + "\n"
    s += "A: " + sgl.gen("car", max_tokens=100, stop='END') + "\n"
    s += "Q: Who would win batman or superman. You must choose ONE character in ONE sentence \n"  
    # s += "A: " + sgl.gen("hero", max_tokens=16, stop='END',)
    s += "A: " + sgl.gen("hero", choices=["batman", "superman"],)


output = test_function.run(test_prompt = "What is the fastest car?")

print(output["car"])
print(output["hero"])

print(output)

Error

Exception in thread Thread-7 (_thread_worker_func):
Traceback (most recent call last):
  File "/home/sr/anaconda3/envs/sglangONLY/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/home/sr/anaconda3/envs/sglangONLY/lib/python3.11/site-packages/ipykernel/ipkernel.py", line 761, in run_closure
    _threading_Thread_run(self)
  File "/home/sr/anaconda3/envs/sglangONLY/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/home/sr/anaconda3/envs/sglangONLY/lib/python3.11/site-packages/sglang/lang/interpreter.py", line 278, in _thread_worker_func
    self._execute(expr)
  File "/home/sr/anaconda3/envs/sglangONLY/lib/python3.11/site-packages/sglang/lang/interpreter.py", line 302, in _execute
    self._execute(x)
  File "/home/sr/anaconda3/envs/sglangONLY/lib/python3.11/site-packages/sglang/lang/interpreter.py", line 299, in _execute
    self._execute_select(other)
  File "/home/sr/anaconda3/envs/sglangONLY/lib/python3.11/site-packages/sglang/lang/interpreter.py", line 376, in _execute_select
    decision, scores = self.backend.select(self, expr.choices, expr.temperature)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/sr/anaconda3/envs/sglangONLY/lib/python3.11/site-packages/sglang/backend/runtime_endpoint.py", line 137, in select
    assert temperature <= 1e-5
           ^^^^^^^^^^^^^^^^^^^
TypeError: '<=' not supported between instances of 'NoneType' and 'float'
 As of now, the Bugatti Veyron Super Sport holds the record for the fastest car in the world, reaching a top speed of 268 mph (431 km/h). However, please note that this record can change as new cars with even greater speeds are developed.

This error also presents itself when I try to run the tool use (calculator + web) example from the documentation

[merrymercy]

This has been fixed in the main branch. Could you try the latest main branch?
We will push a pip version soon.

[psych0v0yager]

Yep the main branch fixed this!

[merrymercy]

@psych0v0yager I pushed another commit and fixed some additional issues of sgl.select. Could you pull the main again or try sglang[all]>=0.1.7?

A tip about using sgalng is "Do not end your generation prompt with a space". This is due to how tokenizers work for the current LLMs. To apply this tip to your example, you can try the following.

@sgl.function
def test_function(s, test_prompt):
    s += "This is a test, I will provide a prompt and answer to the best of your abilities. Once you are done, end with the word 'END'" + "\n"
    s += "Q: " + test_prompt + "\n"
    ### Remove the space after "A:"
    s += "A:" + sgl.gen("car", max_tokens=100, stop='END') + "\n"
    s += "Q: Who would win batman or superman. You must choose ONE character in ONE sentence \n"  
    ### For `choices`, it is okay. You do not need to apply this tip.
    s += "A: " + sgl.gen("hero", choices=["batman", "superman"],)

This sounds tricky. You can learn more about token healing. SGLang has not implemented token healing, but it is on our roadmap. It is currently advised that you avoid ending your generation prompt with a space.

[psych0v0yager]

@merrymercy, thank you for the prompting tip. If you have any more please comment below.

The new pip version worked perfectly, there were zero issues. Thanks for the quick turnaround especially on a weekend.

I am curious, how does sgl (and by extension gudance) control the generation of LLMs. Do you use some kind of system prompt or a special regex parser in the backend to curtail the responses?

Also looking forward to the S-LoRA integration!

[merrymercy]

I will add more documentation about this later.

For choices, it is implemented by computing the normalized log probs of all choices and picking the best one.
For regex, it is implemented by a special regex parser and constrains the logit bias.

sglang does no change any user prompt.