[Roadmap] Intel CPU Roadmap (2025Q4)

[mingfeima]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Previously, we have added optimized CPU backend for SGLang for Xeon with AMX support, enabled Graph Mode with torch.compile and extend model coverage.

In 2025Q4, we will continue optimize CPU backend performance primarily focusing on production deployment:

• Small to medium sized LLMs deployment, e.g. MoE models with activated parameters less than 5B, e.g. Qwen3-Next-80B-A3B.
• OCR models (DeepSeek-OCR), ASR models (whisper), Multimodal models deployment.

General Optimizations

• Graph Mode Improvement: combine pre-compiled batch sizes with explicit user inputs configuration to allow more flexible usage of graph runner for improving overall throughput. @CaoE
• Causal Conv1d Support: add optimized kernels for mamba attention, causal conv1d and flash linear attention, for model support of Qwen3-Next. @mingfeima
• MXFP4 Support: Out of box support of MXFP4 with weight only quantization, dequant MXFP4->BF16 and compute with AMX-BF16 or AVX512-BF16. for model support of GPT-OSS 20B and 120B. @mingfeima
• INT4 Support: 4bit mode is more important for computationally constrained hardware e.g. CPUs. Enable awq INT4 (w4a8), [CPU][INT4] Add AWQ frontend support for CPU  #8225, [CPU][INT4] Add INT4 kernels for CPU  #8226. @jianan-gu
• FP8 KV Cache Support: enable usage of FP8 kv cache, fallback to compute with BF16. @blzheng
• Data Parallel Attention: enable DP MLA. @chunyuan-w

Innovation

• Software pipelining for AMX / AVX512: double buffering for dequant and dot product with AMX / AVX512, increase flops for FP8, MXFP4 and INT4 GEMM and MoE.

User Experience and Testing Enhancement

• Documentation: fulfill documentation, and provide BKMs for optimal configurations for prioritized models @ZailiWang
• Bug Tracking: track bugs, and enable more proxy models in test cases @1pikachu
• Xeon CI: Maintain CI stability, UT enhancement by increasing test case coverage and pass rate @1pikachu

[mingfeima]

Qwen3-Next support related PRs:

sgl-kernel:

• fused_rmsnorm_gated: Add fused_rmsnorm_gated_cpu kernel for CPU to support Qwen3-Next #11577
• chunk_gated_delta_rule: [CPU] Support chunk_gated_delta_rule kernel for Qwen3-Next #12441
• fused_sigmoid_gating_delta_rule_update / fused_gdn_gating: [CPU] add mamba fla kernels for Qwen3-next #12324
• fused_qkvzba_split_reshape_cat: [CPU] add fused_qkvzba_split_reshape_cat kernel for Qwen3-next #12330
• causal_conv1d_fwd / causal_conv1d_update: [CPU] add support for mamba causal conv1d for qwen3-next #12309
• GEMM optimization for out_channels = 1: [CPU] Optimize small oc GEMM for Qwen3-next on CPU #12446
• FP8 GEMM improvement: use faster covnersion from float8_e4m3fn to bfloat16 #12316

python API changes

• python level changes for Q3N: [CPU] Optimize Qwen3-next model on CPU #12525
• CPU graph runner adaption: Add torch.compile support for qwen3-next on CPU #12444

[mingfeima]

We also plan to support multi-modal models such as qwen3-omni and SGLang Diffusion on intel Xeon CPUs:

sgl-kernel:

• flash_attn_varlen_func: [CPU] optimize flash_attn_varlen_func #15708
• prefill optimization for long seqlen: [CPU] improve flash attn kernel performance for extend when seqlen is large #14566
• mrope: [CPU] Add mrope kernel for Qwen3-vl #12531
• apply_rotary_pos_emb_cpu: [CPU] add kernel apply_rotary_pos_emb_cpu for Qwen3-VL and Qwen3-Omni #13121
• Qwen2VLImageProcessorFast: [CPU] Optimize image preprocessor performance for Qwen2VLImageProcessorFast #15168
• conv3d: optimize conv3d used in patch embedding #16040

python API changes:

• Qwen3-Omni and Qwen3-VL: [CPU] Add support for Qwen3-vl and Qwen3-omni #12662