[Router-GO] implement a Go SGLang Router - OpenAI Compatible API Server

[whybeyoung]

Go SGLang Router - OpenAI Compatible API Server

Go SGLang Router is a high-performance OpenAI-compatible API server that communicates with the SGLang backend via gRPC and performs efficient preprocessing and postprocessing through Rust FFI.

CC @slin1237 ， and we need more test for this pr

Features

• ✅ OpenAI API Compatible: Fully compatible with OpenAI Chat Completions API
• ✅ High Performance: Low latency and high throughput using gRPC and Rust FFI
• ✅ Streaming Support: Server-Sent Events (SSE) streaming responses
• ✅ Thread-Safe: Pre-created tokenizer handle, lock-free concurrency
• ✅ Graceful Shutdown: Context cancellation mechanism to avoid resource leaks and panics

Architecture Overview

Important Note: gRPC mode still calls FFI, which is used for:

• Preprocessing: chat_template and tokenization (request phase)
• Postprocessing: token decoding and tool parsing (response phase)

gRPC is only used for communication with the SGLang backend, while input/output processing completely relies on Rust FFI.

┌─────────────────────────────────────────────────────────────────┐
│                        HTTP Client                               │
│                    (OpenAI API Format)                           │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                    FastHTTP Server                               │
│              handlers/chat.go:HandleChatCompletion               │
│              - Parse request JSON                                │
│              - SetBodyStreamWriter (SSE)                        │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│              SGLang Client (client.go)                           │
│         CreateChatCompletionStream(ctx, req)                      │
│         - Wraps gRPC client                                      │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│          gRPC Client (internal/grpc/client_grpc.go)              │
│         CreateChatCompletionStream(ctx, reqJSON)                 │
│                                                                  │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Step 1: FFI Preprocess (Rust FFI)                       │  │
│  │  - ffi.PreprocessChatRequestWithTokenizer()              │  │
│  │  - chat_template application                              │  │
│  │  - tokenization                                           │  │
│  │  - tool constraints generation                            │  │
│  │  Returns: PromptText, TokenIDs, ToolConstraintsJSON,     │  │
│  │           PromptTokens                                   │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                          │
│                       ▼                                          │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Step 2: Build gRPC Request                              │  │
│  │  - Parse request JSON (model, temperature, etc.)        │  │
│  │  - Create proto.GenerateRequest                         │  │
│  │  - Set TokenizedInput (PromptText, TokenIDs)            │  │
│  │  - Set SamplingParams (temperature, top_p, top_k, etc.)  │  │
│  │  - Set Constraints (from ToolConstraintsJSON)            │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                          │
│                       ▼                                          │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Step 3: Create gRPC Stream                              │  │
│  │  - client.Generate(generateReq) → gRPC stream            │  │
│  │  - Connects to SGLang Backend (Rust)                      │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                          │
│                       ▼                                          │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Step 4: Create Converter & BatchPostprocessor          │  │
│  │  - ffi.CreateGrpcResponseConverterWithTokenizer()       │  │
│  │  - Uses preprocessed.PromptTokens for initial count      │  │
│  │  - ffi.NewBatchPostprocessor(batchSize=1, immediate)     │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                          │
│                       ▼                                          │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Step 5: Start readLoop (Background Goroutine)           │  │
│  │  - go grpcStream.readLoop()                               │  │
│  │  - Returns GrpcChatCompletionStream immediately          │  │
│  └────────────────────┬─────────────────────────────────────┘  │
└───────────────────────┼────────────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────────────────┐
│         GrpcChatCompletionStream.readLoop()                     │
│         (Background Goroutine)                                   │
│                                                                  │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Recv() Goroutine (Dedicated)                            │  │
│  │  - Continuously calls stream.Recv()                      │  │
│  │  - Sends results to recvChan (buffered, 2000)          │  │
│  │  - Exits on ctx.Done() or error                          │  │
│  │  - Calls stream.CloseSend() on ctx.Done()               │  │
│  └────────────────────┬─────────────────────────────────────┘  │
│                       │                                          │
│                       ▼                                          │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │  Main Loop                                                │  │
│  │  - Reads from recvChan                                    │  │
│  │  - For each proto.GenerateResponse:                      │  │
│  │    → go processAndSendResponse() (async)                 │  │
│  │      - protoToJSON() converts proto to JSON string        │  │
│  │      - batchPostprocessor.AddChunk(protoJSON)            │  │
│  │        → FFI postprocessing (token decoding, tool parsing)│  │
│  │        → Returns OpenAI-format JSON strings               │  │
│  │      - Sends JSON to resultJSONChan (buffered, 10000)     │  │
│  │      - All operations check ctx.Done() for cancellation  │  │
│  │  - On EOF: flush batch, send remaining results, return  │  │
│  │  - On error: send to errChan (buffered, 100)            │  │
│  │  - defer: cancel ctx, wait goroutines, close channels     │  │
│  └────────────────────┬─────────────────────────────────────┘  │
└───────────────────────┼────────────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────────────────┐
│         resultJSONChan (Buffered Channel, 10000)                 │
│         - Contains OpenAI-format JSON strings                    │
│         - Ready for consumption                                  │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│         ChatCompletionStream.RecvJSON()                          │
│         (client.go:410)                                          │
│         - Direct wrapper: return grpcStream.RecvJSON()           │
│         - No intermediate processing                             │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│         FastHTTP SetBodyStreamWriter                             │
│         (handlers/chat.go:159)                                   │
│         - Loop: stream.RecvJSON() → format SSE → flush         │
│         - Format: "data: {json}\n\n"                           │
│         - Final: "data: [DONE]\n\n"                             │
│         - Immediate flush after each chunk                      │
└────────────────────────────┬────────────────────────────────────┘
                             │
                             ▼
┌─────────────────────────────────────────────────────────────────┐
│                        HTTP Client                               │
│                    (SSE Stream)                                  │
│                    Receives: data: {...}\n\n                    │
└─────────────────────────────────────────────────────────────────┘

Benchmakr vs rust router

## Rust
#Input tokens: 50561
#Output tokens: 25883
Starting warmup with 5 sequences...
Warmup completed with 5 sequences. Starting main benchmark run...

============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    20.0
Max request concurrency:                 20
Successful requests:                     100
Benchmark duration (s):                  37.71
Total input tokens:                      50561
Total input text tokens:                 50561
Total input vision tokens:               0
Total generated tokens:                  25883
Total generated tokens (retokenized):    25599
Request throughput (req/s):              2.65
Input token throughput (tok/s):          1340.75
Output token throughput (tok/s):         686.35
Total token throughput (tok/s):          2027.10
Concurrency:                             18.58
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   7008.05
Median E2E Latency (ms):                 7061.24
---------------Time to First Token----------------
Mean TTFT (ms):                          156.09
Median TTFT (ms):                        133.81
P99 TTFT (ms):                           318.53
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          26.59
Median TPOT (ms):                        26.75
P99 TPOT (ms):                           29.18
---------------Inter-Token Latency----------------
Mean ITL (ms):                           26.71
Median ITL (ms):                         23.61
P95 ITL (ms):                            66.11
P99 ITL (ms):                            115.30
Max ITL (ms):                            201.08
==================================================

## golang
#Input tokens: 50561
#Output tokens: 25883
Starting warmup with 5 sequences...
Warmup completed with 5 sequences. Starting main benchmark run...

============ Serving Benchmark Result ============
Backend:                                 sglang-oai-chat
Traffic request rate:                    20.0
Max request concurrency:                 20
Successful requests:                     100
Benchmark duration (s):                  34.22
Total input tokens:                      50561
Total input text tokens:                 50561
Total input vision tokens:               0
Total generated tokens:                  22970
Total generated tokens (retokenized):    31740
Request throughput (req/s):              2.92
Input token throughput (tok/s):          1477.70
Output token throughput (tok/s):         671.32
Total token throughput (tok/s):          2149.03
Concurrency:                             18.42
----------------End-to-End Latency----------------
Mean E2E Latency (ms):                   6303.33
Median E2E Latency (ms):                 6294.46
---------------Time to First Token----------------
Mean TTFT (ms):                          157.10
Median TTFT (ms):                        149.16
P99 TTFT (ms):                           251.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          26.49
Median TPOT (ms):                        27.15
P99 TPOT (ms):                           28.73
---------------Inter-Token Latency----------------
Mean ITL (ms):                           26.97
Median ITL (ms):                         24.61
P95 ITL (ms):                            52.39
P99 ITL (ms):                            86.52
Max ITL (ms):                            194.55
==================================================

Quick Start

Start Server

./run.sh

The server will start on port :8080.

Usage Example

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/path/to/model",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

Key Design

1. Thread-Safe Tokenizer

• Pre-create TokenizerHandle at startup
• Rust side uses Arc<dyn TokenizerTrait>, thread-safe
• Lock-free concurrency, eliminating lock contention

2. Context Cancellation Mechanism (Graceful Shutdown)

• Use context.Context cancellation mechanism
• In readLoop's defer: cancel context first, then wait for all goroutines to complete, finally close channels
• processAndSendResponse checks ctx.Done() at function start, all select statements include case <-s.ctx.Done()
• Avoids "send on closed channel" panic

3. Cancellable Recv()

• Use dedicated goroutine to execute Recv()
• Pass results through recvChan
• Call CloseSend() when context is cancelled to make Recv() return error

4. Simplified Channel Design

• resultJSONChan: Main data channel (gRPC layer)
• errChan: Error channel (gRPC layer)
• recvChan: Internal communication channel (gRPC layer)
• Removed redundant channels and duplicate reads

Configuration

Channel Buffer Sizes

type ChannelBufferSizes struct {
    ResultJSONChan int // Default: 10000
    ErrChan        int // Default: 100
    RecvChan       int // Default: 2000
}

Timeout Configuration

type Timeouts struct {
    KeepaliveTime    time.Duration // Default: 300s
    KeepaliveTimeout time.Duration // Default: 20s
    CloseTimeout     time.Duration // Default: 5s
}

Performance Optimizations

1. Pre-create Tokenizer: Created at startup to avoid first request latency
2. Lock-Free Concurrency: Tokenizer is thread-safe, no locks needed
3. Lazy Parsing: JSON parsing deferred until needed
4. Direct JSON Passing: RecvJSON() avoids parse/serialize overhead
5. Immediate Batching: batchSize=1, no delay
6. Async Processing: readLoop processes in background, doesn't block request handling
7. Configurable Buffers: Adjust channel sizes based on concurrency needs

File Structure

sgl-model-gateway/bindings/golang/
├── client.go                          # High-level client API
├── internal/
│   ├── grpc/
│   │   └── client_grpc.go            # gRPC client implementation
│   ├── ffi/                          # FFI bindings (Rust)
│   └── proto/                        # Protobuf definitions
└── examples/
    └── oai_server/
        ├── handlers/
        │   └── chat.go               # HTTP request handling
        ├── models/
        │   └── chat.go               # Request/response models
        └── service/
            └── sglang_service.go      # Service layer

Error Handling

Context Cancellation Mechanism

1. Client disconnects → SetBodyStreamWriter detects flush error
2. Cancel streamCtx → readLoop detects ctx.Done()
3. Call stream.CloseSend() → Recv() goroutine returns error
4. readLoop defer executes:
   • Set closed flag
   • Cancel context (if not already cancelled)
   • Wait for all processAndSendResponse goroutines to complete (processWg.Wait())
   • Close all channels (resultJSONChan, errChan, readLoopDone)
5. Clean up resources and exit

Channel Blocking and Race Condition Prevention

• Context cancellation mechanism: All channel sends use select statements with case <-s.ctx.Done()
• Graceful exit: When context is cancelled, all blocking send operations can return immediately
• WaitGroup synchronization: readLoop's defer uses processWg.Wait() to ensure all goroutines complete before closing channels
• Avoid panic: Through context cancellation and WaitGroup synchronization, avoids "send on closed channel" panic

Key Functions

CreateChatCompletionStream

Location: internal/grpc/client_grpc.go:108

• Preprocess request (FFI)
• Build gRPC request
• Create converter and batch processor
• Start readLoop

readLoop

Location: internal/grpc/client_grpc.go:290

• Start Recv() goroutine (continuously calls stream.Recv())
• Process proto responses
• Asynchronously call processAndSendResponse (tracked with processWg)
• Graceful shutdown in defer:
  • Set closed flag
  • Cancel context (if not already cancelled)
  • Wait for all processAndSendResponse goroutines to complete (processWg.Wait())
  • Close all channels (resultJSONChan, errChan, readLoopDone)

processAndSendResponse

Location: internal/grpc/client_grpc.go:379

• Check ctx.Done() at function start, return immediately if cancelled
• Convert proto to JSON
• Call FFI batch processor
• All select statements include case <-s.ctx.Done() for graceful shutdown handling
• Send JSON to channel

RecvJSON

Location:

• internal/grpc/client_grpc.go:412: gRPC layer implementation
• client.go:410: Client wrapper layer
• Read from resultJSONChan
• Directly return JSON string, no parsing needed

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!