[Kernel Slimming] Migrate GPTQ-Marlin repack kernel to JIT by celve · Pull Request #18543 · sgl-project/sglang

celve · 2026-02-10T11:35:53Z

Motivation

See #17865

Modifications

New files:

python/sglang/jit_kernel/csrc/gemm/marlin/gptq_marlin_repack.cuh — JIT-compiled CUDA kernel ported from
sgl-kernel/csrc/gemm/marlin/gptq_marlin_repack.cu
python/sglang/jit_kernel/gptq_marlin_repack.py — Python wrapper with JIT loading and output tensor allocation
python/sglang/jit_kernel/tests/test_gptq_marlin_repack.py
python/sglang/jit_kernel/benchmark/bench_gptq_marlin_repack.py

Modified files:

python/sglang/srt/layers/quantization/gptq.py — Switch gptq_marlin_repack import from sgl_kernel to sglang.jit_kernel
python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa16.py — Same
python/sglang/srt/layers/quantization/marlin_utils_fp8.py — Same

Accuracy Tests

Pass all tests defined in python/sglang/jit_kernel/tests/test_gptq_marlin_repack.py

Benchmarking and Profiling

gptq-marlin-repack-performance:
   size_k  JIT Kernel  AOT Kernel
0   128.0    9.655887    9.664548
1   256.0    9.654155    9.663594
2   512.0    9.723926    9.731931
3  1024.0    9.730906    9.738743
4  2048.0    9.755116    9.765832
5  4096.0   18.020566   18.030832
6  8192.0   34.564093   34.558362

Checklist

Format your code according to the Format code with pre-commit.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
After green CI and required approvals, ask Merge Oncalls to merge.

gemini-code-assist · 2026-02-10T11:36:14Z

Summary of Changes

Hello @celve, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors the gptq_marlin_repack kernel by transitioning its implementation to a JIT compilation approach. This change aims to enhance flexibility and streamline development by allowing kernels to be compiled on demand. The migration involved creating new JIT-specific CUDA and Python components, along with comprehensive testing and benchmarking to validate both correctness and performance, ensuring a seamless transition for downstream users.

Highlights

JIT Kernel Migration: The GPTQ-Marlin repack kernel has been migrated from an Ahead-Of-Time (AOT) compiled sgl_kernel implementation to a Just-In-Time (JIT) compiled sglang.jit_kernel implementation.
New JIT Kernel Components: New files were introduced for the JIT kernel, including the CUDA kernel (gptq_marlin_repack.cuh), its Python wrapper (gptq_marlin_repack.py), dedicated unit tests, and a performance benchmark script.
Import Path Updates: Existing quantization layers (gptq.py, compressed_tensors_wNa16.py, marlin_utils_fp8.py) were updated to import the gptq_marlin_repack function from the new sglang.jit_kernel module.
Performance and Accuracy Verified: Accuracy tests confirm the JIT kernel produces correct results, and benchmarking shows performance comparable to the previous AOT kernel, ensuring no regression in execution speed.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

python/sglang/jit_kernel/benchmark/bench_gptq_marlin_repack.py
- Added a new benchmark script to compare the performance of the JIT and AOT GPTQ-Marlin repack kernels.
python/sglang/jit_kernel/csrc/gemm/marlin/gptq_marlin_repack.cuh
- Added the JIT-compiled CUDA kernel for GPTQ-Marlin repack, ported from the AOT sgl-kernel.
python/sglang/jit_kernel/gptq_marlin_repack.py
- Added a Python wrapper for the JIT-compiled GPTQ-Marlin repack CUDA kernel, handling JIT loading and output tensor allocation.
python/sglang/jit_kernel/tests/test_gptq_marlin_repack.py
- Added new unit tests to verify the correctness of the JIT-compiled GPTQ-Marlin repack kernel against both reference implementations and the AOT kernel.
python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa16.py
- Updated the import statement for gptq_marlin_repack to point to the new JIT-compiled version.
python/sglang/srt/layers/quantization/gptq.py
- Updated the import statement for gptq_marlin_repack to point to the new JIT-compiled version.
python/sglang/srt/layers/quantization/marlin_utils_fp8.py
- Updated the import statement for gptq_marlin_repack to point to the new JIT-compiled version.

Activity

The pull request author, celve, initiated the migration of the GPTQ-Marlin repack kernel to a JIT-compiled approach.
New files for the JIT kernel, including CUDA source, Python wrapper, tests, and benchmarks, were added.
Existing files were modified to switch their dependency from the AOT sgl_kernel to the new JIT sglang.jit_kernel for the repack functionality.
Accuracy tests were performed and passed, ensuring the correctness of the JIT implementation.
Benchmarking results were provided, demonstrating comparable performance between the JIT and AOT kernels.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request successfully migrates the GPTQ-Marlin repack kernel to a Just-In-Time (JIT) compiled version. The changes are well-structured, introducing new files for the JIT kernel, its Python wrapper, comprehensive tests, and benchmarks. The existing codebase is cleanly updated to utilize the new JIT kernel. The tests are particularly robust, verifying the JIT kernel's output against both a CPU reference and the original Ahead-Of-Time (AOT) compiled kernel, which ensures correctness. I've found one minor issue in the benchmark script that I've commented on. Overall, this is a high-quality contribution that improves kernel management flexibility.

gemini-code-assist · 2026-02-10T11:37:46Z

python/sglang/jit_kernel/benchmark/bench_gptq_marlin_repack.py

+        raise ValueError(f"Unknown provider: {provider}")
+
+    ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(fn, quantiles=quantiles)
+    return 1000 * ms, 1000 * max_ms, 1000 * min_ms


The return values for y_min and y_max appear to be swapped. The triton.testing.do_bench_cudagraph function with quantiles=[0.5, 0.2, 0.8] returns (median, p20, p80). The perf_report decorator expects a tuple of (y, y_min, y_max). Consequently, min_ms (the 20th percentile) should be y_min, and max_ms (the 80th percentile) should be y_max. The current implementation will lead to inverted error bars in the benchmark plot.

Suggested change

return 1000 * ms, 1000 * max_ms, 1000 * min_ms

return 1000 * ms, 1000 * min_ms, 1000 * max_ms

celve · 2026-02-10T11:38:17Z

Test MMLU with following scripts:

# server
python3 -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 \
  --port 30000
  
# client
bash benchmark/mmlu/download_data.sh 
python3 benchmark/mmlu/bench_sglang.py

JIT:

100%|██████████████████████████████████████| 14042/14042 [02:21<00:00, 99.09it/s]
subject: abstract_algebra, #q:100, acc: 0.530
subject: anatomy, #q:135, acc: 0.719
subject: astronomy, #q:152, acc: 0.862
subject: business_ethics, #q:100, acc: 0.800
subject: clinical_knowledge, #q:265, acc: 0.770
subject: college_biology, #q:144, acc: 0.833
subject: college_chemistry, #q:100, acc: 0.570
subject: college_computer_science, #q:100, acc: 0.670
subject: college_mathematics, #q:100, acc: 0.470
subject: college_medicine, #q:173, acc: 0.699
subject: college_physics, #q:102, acc: 0.490
subject: computer_security, #q:100, acc: 0.780
subject: conceptual_physics, #q:235, acc: 0.745
subject: econometrics, #q:114, acc: 0.605
subject: electrical_engineering, #q:145, acc: 0.745
subject: elementary_mathematics, #q:378, acc: 0.706
subject: formal_logic, #q:126, acc: 0.563
subject: global_facts, #q:100, acc: 0.470
subject: high_school_biology, #q:310, acc: 0.865
subject: high_school_chemistry, #q:203, acc: 0.626
subject: high_school_computer_science, #q:100, acc: 0.860
subject: high_school_european_history, #q:165, acc: 0.848
subject: high_school_geography, #q:198, acc: 0.864
subject: high_school_government_and_politics, #q:193, acc: 0.943
subject: high_school_macroeconomics, #q:390, acc: 0.787
subject: high_school_mathematics, #q:270, acc: 0.574
subject: high_school_microeconomics, #q:238, acc: 0.903
subject: high_school_physics, #q:151, acc: 0.576
subject: high_school_psychology, #q:545, acc: 0.888
subject: high_school_statistics, #q:216, acc: 0.699
subject: high_school_us_history, #q:204, acc: 0.882
subject: high_school_world_history, #q:237, acc: 0.873
subject: human_aging, #q:223, acc: 0.744
subject: human_sexuality, #q:131, acc: 0.763
subject: international_law, #q:121, acc: 0.835
subject: jurisprudence, #q:108, acc: 0.787
subject: logical_fallacies, #q:163, acc: 0.785
subject: machine_learning, #q:112, acc: 0.589
subject: management, #q:103, acc: 0.874
subject: marketing, #q:234, acc: 0.923
subject: medical_genetics, #q:100, acc: 0.840
subject: miscellaneous, #q:783, acc: 0.849
subject: moral_disputes, #q:346, acc: 0.789
subject: moral_scenarios, #q:895, acc: 0.607
subject: nutrition, #q:306, acc: 0.794
subject: philosophy, #q:311, acc: 0.768
subject: prehistory, #q:324, acc: 0.846
subject: professional_accounting, #q:282, acc: 0.564
subject: professional_law, #q:1534, acc: 0.507
subject: professional_medicine, #q:272, acc: 0.757
subject: professional_psychology, #q:612, acc: 0.773
subject: public_relations, #q:110, acc: 0.691
subject: security_studies, #q:245, acc: 0.771
subject: sociology, #q:201, acc: 0.876
subject: us_foreign_policy, #q:100, acc: 0.900
subject: virology, #q:166, acc: 0.542
subject: world_religions, #q:171, acc: 0.854
Total latency: 141.809
Average accuracy: 0.730

AOT:

100%|██████████████████████████████████████| 14042/14042 [02:23<00:00, 97.78it/s]
subject: abstract_algebra, #q:100, acc: 0.510
subject: anatomy, #q:135, acc: 0.719
subject: astronomy, #q:152, acc: 0.862
subject: business_ethics, #q:100, acc: 0.800
subject: clinical_knowledge, #q:265, acc: 0.770
subject: college_biology, #q:144, acc: 0.833
subject: college_chemistry, #q:100, acc: 0.570
subject: college_computer_science, #q:100, acc: 0.670
subject: college_mathematics, #q:100, acc: 0.470
subject: college_medicine, #q:173, acc: 0.699
subject: college_physics, #q:102, acc: 0.500
subject: computer_security, #q:100, acc: 0.780
subject: conceptual_physics, #q:235, acc: 0.745
subject: econometrics, #q:114, acc: 0.605
subject: electrical_engineering, #q:145, acc: 0.738
subject: elementary_mathematics, #q:378, acc: 0.706
subject: formal_logic, #q:126, acc: 0.563
subject: global_facts, #q:100, acc: 0.470
subject: high_school_biology, #q:310, acc: 0.868
subject: high_school_chemistry, #q:203, acc: 0.626
subject: high_school_computer_science, #q:100, acc: 0.860
subject: high_school_european_history, #q:165, acc: 0.848
subject: high_school_geography, #q:198, acc: 0.864
subject: high_school_government_and_politics, #q:193, acc: 0.943
subject: high_school_macroeconomics, #q:390, acc: 0.787
subject: high_school_mathematics, #q:270, acc: 0.574
subject: high_school_microeconomics, #q:238, acc: 0.903
subject: high_school_physics, #q:151, acc: 0.576
subject: high_school_psychology, #q:545, acc: 0.888
subject: high_school_statistics, #q:216, acc: 0.699
subject: high_school_us_history, #q:204, acc: 0.882
subject: high_school_world_history, #q:237, acc: 0.873
subject: human_aging, #q:223, acc: 0.744
subject: human_sexuality, #q:131, acc: 0.763
subject: international_law, #q:121, acc: 0.835
subject: jurisprudence, #q:108, acc: 0.787
subject: logical_fallacies, #q:163, acc: 0.785
subject: machine_learning, #q:112, acc: 0.589
subject: management, #q:103, acc: 0.874
subject: marketing, #q:234, acc: 0.923
subject: medical_genetics, #q:100, acc: 0.840
subject: miscellaneous, #q:783, acc: 0.848
subject: moral_disputes, #q:346, acc: 0.789
subject: moral_scenarios, #q:895, acc: 0.607
subject: nutrition, #q:306, acc: 0.794
subject: philosophy, #q:311, acc: 0.772
subject: prehistory, #q:324, acc: 0.843
subject: professional_accounting, #q:282, acc: 0.564
subject: professional_law, #q:1534, acc: 0.505
subject: professional_medicine, #q:272, acc: 0.757
subject: professional_psychology, #q:612, acc: 0.773
subject: public_relations, #q:110, acc: 0.691
subject: security_studies, #q:245, acc: 0.771
subject: sociology, #q:201, acc: 0.876
subject: us_foreign_policy, #q:100, acc: 0.900
subject: virology, #q:166, acc: 0.542
subject: world_religions, #q:171, acc: 0.854
Total latency: 143.694
Average accuracy: 0.730

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0183bd0448

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-02-10T11:40:28Z

python/sglang/jit_kernel/csrc/gemm/marlin/gptq_marlin_repack.cuh

+
+  // Get ptrs
+  uint32_t const* b_q_weight_ptr = reinterpret_cast<uint32_t const*>(b_q_weight.data_ptr());
+  uint32_t const* perm_ptr = reinterpret_cast<uint32_t const*>(perm.data_ptr());


Validate perm tensor before launching repack kernel

The JIT port no longer validates perm (dtype, contiguity, or CUDA placement) before reinterpreting it as uint32_t*; when has_perm is true and callers pass common torch.argsort output (int64) or a CPU/non-contiguous tensor, the kernel will read wrong indices or invalid memory, producing corrupted repacked weights or a device fault. The previous AOT implementation explicitly rejected these inputs, so this is a regression in input safety for act-order repack paths.

Useful? React with 👍 / 👎.

Copilot

Pull request overview

Migrates the GPTQ-Marlin repack CUDA kernel from the AOT sgl_kernel extension to the sglang.jit_kernel JIT compilation path, and updates quantization callsites to use the new JIT implementation to support the kernel wheel slimming plan.

Changes:

Add a JIT-compiled gptq_marlin_repack CUDA kernel and Python wrapper under sglang.jit_kernel.
Update GPTQ/Marlin quantization codepaths to import gptq_marlin_repack from sglang.jit_kernel instead of sgl_kernel.
Add a unit test and a Triton-based benchmark for the new JIT repack kernel.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
python/sglang/srt/layers/quantization/marlin_utils_fp8.py	Switch repack import to the JIT kernel module.
python/sglang/srt/layers/quantization/gptq.py	Switch repack import to the JIT kernel module.
python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_wNa16.py	Switch repack import to the JIT kernel module.
python/sglang/jit_kernel/gptq_marlin_repack.py	New Python wrapper that JIT-loads the kernel and allocates the output tensor.
python/sglang/jit_kernel/csrc/gemm/marlin/gptq_marlin_repack.cuh	New JIT CUDA implementation and host entrypoint for GPTQ-Marlin repack.
python/sglang/jit_kernel/tests/test_gptq_marlin_repack.py	New unit test covering correctness of the JIT repack output.
python/sglang/jit_kernel/benchmark/bench_gptq_marlin_repack.py	New benchmark comparing JIT vs AOT when available.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-10T11:48:10Z

python/sglang/jit_kernel/csrc/gemm/marlin/gptq_marlin_repack.cuh

+  bqw_dim1.set_value(size_n);
+  auto device_ = SymbolicDevice{};
+  device_.set_options<kDLCUDA>();
+  TensorMatcher({bqw_dim0, bqw_dim1}).with_dtype<int32_t>().with_device(device_).verify(b_q_weight);


The kernel assumes b_q_weight is contiguous row-major (it does b_q_weight_ptr[row * size_n + col]), but the current TensorMatcher checks only shape/dtype/device. Please also enforce expected strides/contiguity for b_q_weight to prevent incorrect results when a non-contiguous view is passed.

Suggested change

TensorMatcher({bqw_dim0, bqw_dim1}).with_dtype<int32_t>().with_device(device_).verify(b_q_weight);

TensorMatcher({bqw_dim0, bqw_dim1})

.with_dtype<int32_t>()

.with_device(device_)

.with_contiguous()

.verify(b_q_weight);

Copilot · 2026-02-10T11:48:11Z

python/sglang/jit_kernel/csrc/gemm/marlin/gptq_marlin_repack.cuh

+  out_dim0.set_value(size_k / device::marlin::tile_size);
+  out_dim1.set_value(size_n * device::marlin::tile_size / pack_factor);
+  TensorMatcher({out_dim0, out_dim1}).with_dtype<int32_t>().with_device(device_).verify(out);
+


Similarly, out is written via raw pointer arithmetic assuming a contiguous row-major layout. Consider enforcing expected strides/contiguity for out (or requiring contiguous) in the validation to avoid silent miswrites if a non-contiguous tensor is ever passed in.

Suggested change

// Enforce that `out` is laid out contiguously in row-major order.

// The kernel writes using raw pointer arithmetic assuming:

// stride(1) == 1

// stride(0) == out_dim1 (i.e., contiguous rows)

auto out_stride0 = out.stride(0);

auto out_stride1 = out.stride(1);

RuntimeCheck(

out_stride1 == 1 && out_stride0 == out_dim1.value(),

"Expected `out` to be a contiguous row-major tensor with shape (",

out_dim0.value(),

", ",

out_dim1.value(),

"), but got strides (",

out_stride0,

", ",

out_stride1,

").");

Copilot · 2026-02-10T11:48:11Z

python/sglang/jit_kernel/csrc/gemm/marlin/gptq_marlin_repack.cuh

+    int64_t size_k,
+    int64_t size_n,
+    int64_t num_bits) {


This function takes size_k/size_n as int64_t, but the CUDA kernel parameters are typed as int. Add an explicit bounds check (or change the kernel signature) to prevent silent truncation for large sizes, which could lead to out-of-bounds accesses.

Copilot · 2026-02-10T11:48:11Z

python/sglang/jit_kernel/tests/test_gptq_marlin_repack.py

+import pytest
+import torch
+from sgl_kernel import gptq_marlin_repack as aot_gptq_marlin_repack
+from sgl_kernel.scalar_type import scalar_types


The test imports sgl_kernel unconditionally for the AOT reference, so it will fail with ImportError when sgl_kernel isn’t installed. Consider try/except ImportError and skipping only the JIT-vs-AOT bitwise comparison when AOT isn’t available (the CPU reference check can still run).

Copilot · 2026-02-10T11:48:11Z

python/sglang/jit_kernel/csrc/gemm/marlin/gptq_marlin_repack.cuh

+  // Detect if there is act_order
+  bool has_perm = perm.size(0) != 0;
+
+  // Get ptrs
+  uint32_t const* b_q_weight_ptr = reinterpret_cast<uint32_t const*>(b_q_weight.data_ptr());


has_perm is inferred from perm.size(0), but when it’s non-empty the code never verifies perm’s device/dtype/contiguity or that its length matches size_k. If a wrong tensor is passed, the kernel can read invalid memory. Add TensorMatcher/RuntimeChecks for perm and enforce perm.size(0) == 0 || perm.size(0) == size_k before launch.

BBuf

LGTM.

BBuf · 2026-02-10T12:54:57Z

Fix lint

celve · 2026-02-11T01:48:20Z

Fix lint

Done

BBuf · 2026-02-12T01:05:52Z

/rerun-failed-ci

BBuf · 2026-02-13T04:35:26Z

/rerun-failed-ci

BBuf · 2026-02-13T07:50:37Z

/tag-and-rerun-ci

BBuf · 2026-02-13T14:29:12Z

https://github.com/sgl-project/sglang/actions/runs/21978982592/job/63513329595?pr=18543

…ct#18543) Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>

wip: gptq repack

0183bd0

Copilot AI review requested due to automatic review settings February 10, 2026 11:35

celve requested review from AniZpZ, BBuf, DarkSharpness, Edwardf0t1, FlamingoPg and ch-wan as code owners February 10, 2026 11:35

Copilot started reviewing on behalf of celve February 10, 2026 11:36 View session

gemini-code-assist bot reviewed Feb 10, 2026

View reviewed changes

chatgpt-codex-connector bot reviewed Feb 10, 2026

View reviewed changes

celve changed the title ~~[Kernel] Migrate GPTQ-Marlin repack kernel to JIT~~ [Kernel Slimming] Migrate GPTQ-Marlin repack kernel to JIT Feb 10, 2026

Copilot AI reviewed Feb 10, 2026

View reviewed changes

BBuf approved these changes Feb 10, 2026

View reviewed changes

wip: lint

df29fa5

BBuf added 2 commits February 12, 2026 13:56

Merge branch 'main' into jit-gptq-marlin-repack

a065314

Merge branch 'main' into jit-gptq-marlin-repack

eea0e94

BBuf added 2 commits February 13, 2026 15:27

Merge branch 'main' into jit-gptq-marlin-repack

3acaa13

Merge branch 'main' into jit-gptq-marlin-repack

e29eff1

github-actions bot added the run-ci label Feb 13, 2026

BBuf merged commit 0012d6a into sgl-project:main Feb 13, 2026
187 of 203 checks passed

BBuf mentioned this pull request Feb 13, 2026

[Feature] sgl-kernel wheel slimming plan tracking #17865

Open

74 tasks

Johnsonms pushed a commit to Johnsonms/sglang that referenced this pull request Feb 14, 2026

[Kernel Slimming] Migrate GPTQ-Marlin repack kernel to JIT (sgl-proje…

af4ff57

…ct#18543) Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>

	return 1000 * ms, 1000 * max_ms, 1000 * min_ms
	return 1000 * ms, 1000 * min_ms, 1000 * max_ms

+  // Enforce that `out` is laid out contiguously in row-major order.
+  // The kernel writes using raw pointer arithmetic assuming:
+  //   stride(1) == 1
+  //   stride(0) == out_dim1 (i.e., contiguous rows)
+  auto out_stride0 = out.stride(0);
+  auto out_stride1 = out.stride(1);
+  RuntimeCheck(
+      out_stride1 == 1 && out_stride0 == out_dim1.value(),
+      "Expected `out` to be a contiguous row-major tensor with shape (",
+      out_dim0.value(),
+      ", ",
+      out_dim1.value(),
+      "), but got strides (",
+      out_stride0,
+      ", ",
+      out_stride1,
+      ").");

Comments

Conversation

celve commented Feb 10, 2026

Motivation

Modifications

Accuracy Tests

Benchmarking and Profiling

Checklist

Review Process

Uh oh!

gemini-code-assist bot commented Feb 10, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

celve commented Feb 10, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

BBuf left a comment

Choose a reason for hiding this comment

Uh oh!

BBuf commented Feb 10, 2026

Uh oh!

celve commented Feb 11, 2026

Uh oh!

BBuf commented Feb 12, 2026

Uh oh!

BBuf commented Feb 13, 2026

Uh oh!

BBuf commented Feb 13, 2026

Uh oh!

BBuf commented Feb 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants