[model-gateway] Add tokenize/detokenize HTTP endpoints and tokenizer management

[slin1237]

This adds HTTP endpoints for tokenization operations similar to SGLang Python:

• POST /v1/tokenize - tokenize text into token IDs
• POST /v1/detokenize - convert token IDs back to text
• POST /v1/tokenizers - add a tokenizer (async via job queue)
• GET /v1/tokenizers - list all registered tokenizers
• GET /v1/tokenizers/{uuid} - get tokenizer info
• GET /v1/tokenizers/{uuid}/status - get tokenizer loading status
• DELETE /v1/tokenizers/{uuid} - remove a tokenizer

Key implementation details:

• Uses workflow pattern for tokenizer registration to support async loading from HuggingFace (similar to worker registration)
• Tokenizer deletion is synchronous since it's a simple HashMap remove with no external resources to clean up
• Supports both single text and batch tokenization/detokenization
• Returns job_id immediately for add_tokenizer, status can be polled

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!