[OAI Server Refactor] Implement Embeddings, Scoring, and Rerank

[CatherineSue]

Points: 3-4 days

Description: Implement endpoints for embeddings, scoring, and rerank.

Deliverables:

• Create serving_embedding.py and serving_score.py
• Refactor /v1/embeddings and /v1/score from adapter.py
• Mount endpoints in api_server.py

Testing:

• Add UTs if possible

[woodx9]

@CatherineSue Can I take this one? I also support rerank endpoint #6058, but it's not merge yet.

[CatherineSue]

@woodx9 Sure! Help is always appreciated! @zhyncs will add you to the refactor channel.

[CatherineSue]

TODOs in serving_embedding.py in #7167

1. Request preprocessing

Task 1: Remove Batch Requests

• Remove the batch requests process.
  Specifically:
  

  sglang/python/sglang/srt/entrypoints/openai/serving_embedding.py

  Lines 134 to 169 in 70c471a

  	else:
  	# Handle batch requests
  	if len(prompts) > 0:
  	# Validate that all prompts have the same type
  	first_prompt = prompts[0]
  	first_type = type(first_prompt)
  	for i, prompt in enumerate(prompts[1:], 1):
  	if type(prompt) != first_type:
  	raise AssertionError(
  	f"All prompts in batch must have the same type, but prompt at index {i} has different type"
  	)
  	if isinstance(first_prompt, str):
  	# Batch of strings
  	prompt_kwargs = {"text": prompts}
  	elif isinstance(first_prompt, list):
  	if len(first_prompt) > 0 and isinstance(first_prompt[0], str):
  	# Batch of lists of strings
  	prompt_kwargs = {"text": prompts}
  	elif len(first_prompt) > 0 and isinstance(
  	first_prompt[0], MultimodalEmbeddingInput
  	):
  	# Handle multimodal batch requests
  	raise NotImplementedError(
  	"Multiple requests with multimodal inputs are not supported yet"
  	)
  	else:
  	# Batch of token ID lists
  	prompt_kwargs = {"input_ids": prompts}
  	else:
  	# Other types
  	prompt_kwargs = {"input_ids": prompts}
  	else:
  	prompt_kwargs = {"input_ids": prompts}
  	# Use the passed request_ids for batch requests
  	final_request_id = request_ids

• We won't support List[EmbeddingRequest], only EmbeddingRequest.

• Also check ifrequest_id can be removed in this way as well.

Task 2: Check _validate_request

• Make sure the validation of list type input in _validate_request is consistent with adapter.py.

• Code block in serving_embedding.py:
  

  sglang/python/sglang/srt/entrypoints/openai/serving_embedding.py

  Lines 35 to 70 in 70c471a

  	# Handle list inputs
  	if isinstance(input, list):
  	if len(input) == 0:
  	return "Input cannot be empty"
  	# Check first element to determine type
  	first_item = input[0]
  	if isinstance(first_item, str):
  	# List of strings
  	for i, item in enumerate(input):
  	if not isinstance(item, str):
  	return f"All items in input list must be strings"
  	if not item.strip():
  	return f"Input at index {i} cannot be empty or whitespace only"
  	elif isinstance(first_item, int):
  	# List of integers (token IDs)
  	for i, item in enumerate(input):
  	if not isinstance(item, int):
  	return f"All items in input list must be integers"
  	if item < 0:
  	return f"Token ID at index {i} must be non-negative"
  	elif isinstance(first_item, list):
  	# List of lists (multiple token sequences)
  	for i, item in enumerate(input):
  	if not isinstance(item, list):
  	return f"Input at index {i} must be a list"
  	if not item:
  	return f"Input at index {i} cannot be empty"
  	if not all(isinstance(token, int) for token in item):
  	return f"Input at index {i} must contain only integers"
  	if any(token < 0 for token in item):
  	return f"Input at index {i} contains negative token IDs"
  	# Note: MultimodalEmbeddingInput validation would be handled by Pydantic
  	return None

• adapter.py:
  

  sglang/python/sglang/srt/openai_api/adapter.py

  Lines 196 to 204 in ca92911

  	# Check for various invalid list cases: [], [""], [" "], [[]]
  	elif isinstance(prompt, list):
  	is_invalid = not prompt or (
  	len(prompt) == 1
  	and (
  	(isinstance(prompt[0], str) and not prompt[0].strip())
  	or (isinstance(prompt[0], list) and not prompt[0])
  	)
  	)

No extra validation for now.

2. Non-streaming Response Generation

Task 1: Function init

• Remove model_path parameter, directly get from self.tokenizer_manager. Ref:

  sglang/python/sglang/srt/entrypoints/openai/serving_embedding.py

  Lines 197 to 200 in 70c471a

  	response = self._build_embedding_response(
  	ret, self.tokenizer_manager.model_path
  	)
  	return response

• During non-streaming, I think the current tokenizer_manager will always return a List type of ret.
  Check if it is ok to remove the following line at

  sglang/python/sglang/srt/entrypoints/openai/serving_embedding.py

  Lines 194 to 195 in 70c471a

  	if not isinstance(ret, list):
  	ret = [ret]

[woodx9]

will support embedding refactor first and then socre and rerank later this weekend.

[woodx9]

Request Preprocessing

Task 1: Remove Batch Requests

• Only support EmbeddingRequest (done)
• Removed request_id from EmbeddingRequest (done)
  Note: EmbeddingReqInput will auto-generate rids for both single and multi-batch inputs.
  It's better not to assign rid manually for multi-batches.
  Reference: io_struct.py#L546-L562

Task 2: Check _validate_request

• Confirmed: the current _validate_request logic is more robust than the one in adapter (done)

---

Non-streaming Response Generation

Task 1: Function Init

• Removed model_path parameter; now obtained directly from self.tokenizer_manager (done)
• List-wrapping logic not removed (not done)
  Note: For single requests, ret is an object instead of a list.
  Keeping the list-wrapping logic ensures correct handling of both single and multi-batch inputs.

[CatherineSue]

Completed in PRs: #7322 #7399