[Feature] request for "classify" method for sglang router

[yyDing1]

Checklist

• If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
• Please use English. Otherwise, it will be closed.

Motivation

In RL, we may launch a router to dispatch reward computation requests for both generative rewards (via /generate) and discriminative rewards (via /classify). I note that the "classify" api has not been implemented in sglang router (https://docs.sglang.io/advanced_features/router.html#api-surface). Hope future release will include this feat.

In addition to this, I want to report a bug that in the latest sglang router, the router fails to handle "v1/embeddings" requests after the servers sleep and then wake up.

# step 1: Launch multiple workers and one router
# step 2: Sleep servers and then wake up them
# step 3: Post "v1/embeddings" requests to router

For sglang-router==0.2.4
curl http://{router_address}/v1/models
{"object":"list","data":[{"id":"unknown","object":"model","owned_by":"local"}]}
and the reward computation also fails

For sglang-router==0.2.2
all behaviors are correct

@slin1237 @zhaochenyang20

Related resources

No response

[zhaochenyang20]

@yyDing1 Thanks, Really make sense. We shall find someone to add one.

[alphabetc1]

@yyDing1 Thanks, Really make sense. We shall find someone to add one.

@zhaochenyang20 Hi, I’m new to sglang and would like to take this issue. Is it okay if I work on it?

[zhaochenyang20]

@yyDing1 Thanks, Really make sense. We shall find someone to add one.

@zhaochenyang20 Hi, I’m new to sglang and would like to take this issue. Is it okay if I work on it?

It would be great. Please do it

[majiayu000]

Hi! I noticed that the /v1/classify API is already implemented in the router since PR #11618 (October 2025).

The route is available at /v1/classify in sgl-model-gateway/src/server.rs:656.

Regarding the embeddings bug after sleep/wake up - this would require further investigation. Could you provide more details about:

1. The exact steps to reproduce
2. Any error logs from the router
3. The worker configuration being used

This would help track down the root cause.

[yyDing1]

The minimal reproduction scripts:

My env is sglang==0.5.5 (released in Nov 2025), sglang-router==0.2.4 (latest)

# init two workers and one router
model="/path/to/Skywork-Reward-V2-Llama-3.2-1B"
CUDA_VISIBLE_DEVICES=0 nohup python -m sglang.launch_server --model-path $model --port 20001 > server1.out &
CUDA_VISIBLE_DEVICES=1 nohup python -m sglang.launch_server --model-path $model --port 20002 > server2.out &
nohup python -m sglang_router.launch_router --worker-urls http://127.0.0.1:20001 http://127.0.0.1:20002 --policy cache_aware --port 30000 > router.out &
curl -s http://127.0.0.1:30000/v1/embeddings -H "Content-Type: application/json" -d '{"model": "$model", "input": "hello"}'
# > {"data":[{"embedding":[-1.3359375],"index":0,"object":"embedding"}],"model":"xxx","object":"list","usage":{"prompt_tokens":2,"total_tokens":2,"completion_tokens":0,"prompt_tokens_details":null,"reasoning_tokens":0}}
# This is OK
curl -s http://127.0.0.1:30000/v1/classify -H "Content-Type: application/json" -d '{"model": "$model", "input": "hello"}'
# > {"id":"classify-29bf3f49572c45a19bef64761a2823a8","object":"list","created":1766054106,"model":"xxx","data":[{"index":0,"label":"LABEL_0","probs":[1.0],"num_classes":1}],"usage":{"prompt_tokens":2,"total_tokens":2,"completion_tokens":0,"prompt_tokens_details":null,"reasoning_tokens":0}}
# This is wrong

This demonstrates that there is something wrong with v1/classify.

Then I sleep the servers:

curl http://127.0.0.1:20001/release_memory_occupation -H "Content-Type: application/json" -d '{"tags": ["kv_cache", "weights"]}'
curl http://127.0.0.1:20002/release_memory_occupation -H "Content-Type: application/json" -d '{"tags": ["kv_cache", "weights"]}'
curl http://127.0.0.1:20001/resume_memory_occupation -H "Content-Type: application/json" -d '{"tags": ["kv_cache", "weights"]}'
curl http://127.0.0.1:20002/resume_memory_occupation -H "Content-Type: application/json" -d '{"tags": ["kv_cache", "weights"]}'

I observe timeout info in router logs, like

# 2025-12-18 11:36:26  WARN sgl_model_gateway::core::worker: /home/runner/work/sglang/sglang/src/core/worker.rs:807: HTTP health check failed for http://127.0.0.1:20002: reqwest::Error { kind: Request, url: "http://127.0.0.1:20002/health", source: TimedOut }
# 2025-12-18 11:36:26  WARN sgl_model_gateway::core::worker: /home/runner/work/sglang/sglang/src/core/worker.rs:807: HTTP health check failed for http://127.0.0.1:20001: reqwest::Error { kind: Request, url: "http://127.0.0.1:20001/health", source: TimedOut }

And the corresponding timeout logs in server logs:

[2025-12-18 19:36:41] Health check failed. Server couldn't get a response from detokenizer for last 20 seconds. tic start time: 19:36:21. last_heartbeat time: 19:27:53

But I'm confused that the gpu memory is released just a bit (~100MB) after the calling of resume_memory_occupation. Please correct me if I make some mistakes.

[alphabetc1]

hi, @yyDing1
The model name will always be set to unknown（this is your first question, i guess?）. This is caused by a version compatibility issue between sglang and the router. For older versions, the router obtains model name via /get_server_info, whereas for newer versions, it uses /server_info (which is not implemented in older sglang), resulting in the model being fixed as unknown. I’ll submit a fix for this later.

# sglang 0.5.5+router 0.2.2, ok
# curl http://127.0.0.1:30000/v1/models
# {"models":["/root/models/Skywork-Reward-V2-Llama-3.2-1B"]}
INFO:     127.0.0.1:38382 - "GET /health HTTP/1.1" 200 OK
INFO:     127.0.0.1:38400 - "GET /get_server_info HTTP/1.1" 200 OK

# sglang 0.5.5+router 0.2.4, error: 
# curl http://127.0.0.1:30000/v1/models
# {"object":"list","data":[{"id":"unknown","object":"model","owned_by":"local"}]}
INFO:     127.0.0.1:45944 - "GET /server_info HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:45944 - "GET /model_info HTTP/1.1" 404 Not Found

As for the "/v1/classify is wrong" and the “reward computation also fails” problem, could you provide a comparison example between the correct output and the erroneous output?

[yyDing1]

Thanks for your clarification.

I further search the related doc and find that /v1/classify and /classify actually do different things.

The /classify endpoint in SGLang was originally designed for reward models but now supports all non-generative models. Our /v1/classify endpoint provides a standardized vLLM-compatible interface for classification tasks.

See https://docs.sglang.io/supported_models/classify_models.html#reward-models-single-score). So there may have been misunderstandings. Sorry for the confusion. I mean "/classify" has been supported for the sgl-server yet not supported for sgl-router. @majiayu000

This is also reflected in the results.

# post "classify" to server
curl -s http://127.0.0.1:20001/classify -H "Content-Type: application/json" -d '{"model": "$model", "text": "hello"}'
# > "embedding":[-1.359375], this is correct value, but "classify" is not supported for router

# post "v1/classify" to router
curl -s http://127.0.0.1:30000/v1/classify -H "Content-Type: application/json" -d '{"model": "$model", "input": "hello"}'
# > "probs":[1.0], this return value is different. "v1/classify" can not handle the reward computation requests.

I also note that v1/embeddings is supported for router and seems to get similar results as "classify"

curl -s http://127.0.0.1:30000/v1/embeddings -H "Content-Type: application/json" -d '{"model": "$model", "input": "hello"}'
# > "embedding":[-1.328125]
curl -s http://127.0.0.1:20001/classify -H "Content-Type: application/json" -d '{"model": "$model", "text": "hello"}'
# > "embedding":[-1.328125]

So it seems to be a usable substitute for "classify" for router?

I'm reproducing the sleep&wakeup issue in a minimal reproduction scripts, and will provide one soon.

[zhaochenyang20]

@alphabetc1 @yyDing1 would you guys like to connect my wechat for discussion? please add this:

LoveDeathAndLLM

[alphabetc1]

@yyDing1
The version compatibility issue has already been fixed in this PR:
#15413.
Could you try to reproduce the problem using router v0.3.0?

/classify directly reuses EmbeddingReqInput (the comment even says “reward model request… parameters and return are the same as embedding”). Meanwhile, model-gateway only exposes the OpenAI v1 namespace for classify, so it only mounts /v1/classify.
If the old /classify meets your needs, /v1/embeddings should also work for you.