[OAI Server Refactor] [ChatCompletions & Completions] Support Return Hidden States

[CatherineSue]

Points: 2-3 days

Description: Support return hidden states in serving_chat.py and serving_completions.py

Deliverables:

• Complete task below
• Add UTs

Task: Missing hidden states **

Current streaming logic in adapter.py#L691-699:

sglang/python/sglang/srt/openai_api/adapter.py

Lines 691 to 699 in ca92911

	hidden_states = None
	if isinstance(request, list) and request[idx].return_hidden_states:
	hidden_states = ret_item["meta_info"].get("hidden_states", None)
	elif (not isinstance(request, list)) and request.return_hidden_states:
	hidden_states = ret_item["meta_info"].get("hidden_states", None)
	if hidden_states is not None:
	hidden_states = (
	hidden_states[-1] if hidden_states and len(hidden_states) > 1 else []
	)

Current logic in adapter.py#L1370-L1382:

sglang/python/sglang/srt/openai_api/adapter.py

Lines 1370 to 1382 in ca92911

	if isinstance(request, list) and request[idx].return_hidden_states:
	include_hidden_states = True
	elif not isinstance(request, list) and request.return_hidden_states:
	include_hidden_states = True
	else:
	include_hidden_states = False
	if include_hidden_states and ret_item["meta_info"].get("hidden_states", None):
	hidden_states = ret_item["meta_info"]["hidden_states"]
	hidden_states = (
	hidden_states[-1] if hidden_states and len(hidden_states) > 1 else []
	)
	else:
	hidden_states = None

• Need to support it in serving_chat.py and serving_completions.py .

[CatherineSue]

cc @key4ng

[CatherineSue]

Completed