[Roadmap] Quantization Modifications

[TamirBaydasov]

1. Quantization refactor

Background

Scheme structure

Currently, quantization methods are mostly implemented in 2 ways: with schemes structure, like compressed_tensors and quark, and without, like modelslim for NPU or AWQ. While foregoing scheme structure can be ok if we only load 1 format, having ability to load many of them without specific logic will overload get_quant_method function. Overall, the more quant methods get supported, the bigger each respective file gets.

Weight loading and inference

The weight creation and inference code is currently implemented in the same class, but same inference code can be utilized by different frameworks.

Motivation

Support more scheme structures

The key improvement of following the scheme structure is that it is much easier to maintain and update, allowing for easier implementation and review process.
Below is the example of proposed scheme structure change:

Split weight loading and inference

Quant config, weight creation and kernel call logic should be clearly separated to allow different frameworks to use the same kernel if it fits. This will avoid code duplication and thus increase code readability and avoid circular imports. Our end goal is to create a unified and simpler structure for quantization functionality. The main source of inspiration for refactoring ideas came from compressed-tensors scheme structure.
Below are the image examples of proposed change for AWQ:

• NPU specific refactoring and format support [Feature] Ascend NPU quantization refactoring & more quantization formats support #14424

• Compressed-Tensors, ModelSlim, Quark MoE schemes

  • [2/N] Quantization Refactor: Compressed tensors MoE schemes #17503
  • [3/N] Quantization Refactor: ModelSlim MoE schemes #17993
  • [4/N] Quantization Refactor: Quark MoE schemes #18252
  • Look into

    sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py

    Line 668 in f8636fb

    # compressed-tensors checkpoints with packed weights are stored flipped

    for possible improvement

• Support schemes for AWQ, GPTQ, Auto Round, GGUF

• Kernel call and weight init split

2. Non-Linear Module & Communication Quantization

Objective: Optimize components beyond standard linear layers to further improve performance.

• Attention

  • MLA Quantization @hammersam

  • GQA/MHA Quantization

• Improved KV Cache Quantization

  • feat: Add FP8 KV cache support for Triton attention backend #18882 @zack041
  • Support int8 kv cache for NPU @TamirBaydasov

• Communication Quantization

  • AllReduce @m8ngotree

3. New formats support

• NVFP4 Quantization support
  Roadmap: SGLang Nvidia Collaboration Roadmap (2026 Q1) #17130

• Improved AutoQuantize
  Roadmap: SGLang Nvidia Collaboration Roadmap (2026 Q1) #17130

• FP4 KV-Cache Support
  Roadmap: SGLang Nvidia Collaboration Roadmap (2026 Q1) #17130

• mxfp8 support @zianglih Add mxfp8 support for online quantization, Triton dense linear, and CUTLASS MoE #17449

• Online Rotation (for FlatQuant and etc.)

• Vector quantization (for QuIP#, AQLM, VPTQ)

[zianglih]

Additionally I am working on adding mxfp8 support: #17449

[rainj-me]

Just quick catch up, we currently has mutiple places to track the kernels. It is totally fine to create a hardware backend folder for the quantization kernels, but maybe we should also organize all other kernels the same way. Or, maybe we fit the quantization kernel to existing kernel folder.

[TamirBaydasov]

Just quick catch up, we currently has mutiple places to track the kernels. It is totally fine to create a hardware backend folder for the quantization kernels, but maybe we should also organize all other kernels the same way. Or, maybe we fit the quantization kernel to existing kernel folder.

Currently, hardware_backend/npu folder already contains kernels not only for quantization, but for moe and attention as well. I would like to do the same for gpu. Quantization part is a starting point we are most familiar with. We will discuss whether it is a feasible task to do that for the whole gpu part and get back to you.

[nvpohanh]

I am new here, but where would ModelOpt quantized models (like per-tensor w8a8 or NVFP4) sit in these pictures?

[TamirBaydasov]

I am new here, but where would ModelOpt quantized models (like per-tensor w8a8 or NVFP4) sit in these pictures?

Since ModelOpt supports 2 quantization formats right now, with possibility of more in the future, our current view is that it should be a scheme-like structure, like in our first suggestion.
Our 2nd suggestion about kernel call split will be applied to every quantization algorithm.

What that means for ModelOpt is it is going to be a scheme structure with apply methods in each scheme calling its specific kernel from hardware_backend/gpu/quantization/...

[rainj-me]

Just quick catch up, we currently has mutiple places to track the kernels. It is totally fine to create a hardware backend folder for the quantization kernels, but maybe we should also organize all other kernels the same way. Or, maybe we fit the quantization kernel to existing kernel folder.

Currently, hardware_backend/npu folder already contains kernels not only for quantization, but for moe and attention as well. I would like to do the same for gpu. Quantization part is a starting point we are most familiar with. We will discuss whether it is a feasible task to do that for the whole gpu part and get back to you.

I believe the hardware_backend folder is mostly for non-cuda backend kernels to avoid the npu/rocm etc. kernels make too many branch code. From the proposed design, quantization with cuda-kernel would also put in the hardware_backend/gpu/quantization, right?

[TamirBaydasov]

Just quick catch up, we currently has mutiple places to track the kernels. It is totally fine to create a hardware backend folder for the quantization kernels, but maybe we should also organize all other kernels the same way. Or, maybe we fit the quantization kernel to existing kernel folder.

Currently, hardware_backend/npu folder already contains kernels not only for quantization, but for moe and attention as well. I would like to do the same for gpu. Quantization part is a starting point we are most familiar with. We will discuss whether it is a feasible task to do that for the whole gpu part and get back to you.

I believe the hardware_backend folder is mostly for non-cuda backend kernels to avoid the npu/rocm etc. kernels make too many branch code. From the proposed design, quantization with cuda-kernel would also put in the hardware_backend/gpu/quantization, right?

Yes, you are right.

[TamirBaydasov]

It is totally fine to create a hardware backend folder for the quantization kernels, but maybe we should also organize all other kernels the same way.

Hi, I'd like to follow up on this proposal. The process of moving all cuda-related kernels into hardware_backend structure is something we would like to work towards, but it heavily exceeds the scope of this issue. It also should be aligned with HAL creation issue (#15299) in my opinion , where we can attract more engineers to help.

[m8ngotree]

Hi, I am interested in and currently working on communication quantization. I am currently working on a PR for quantized AllReduce.

[TamirBaydasov]

Hi, I am interested in and currently working on communication quantization. I am currently working on a PR for quantized AllReduce.

Hi! Thank you for taking an interest. Added to roadmap. When you make a PR, send a link here and I'll add it as well.