[10/N] MoE Refactor: reorganize deepgemm runner in DeepEPMoE

[ch-wan]

Motivation

Deprecate forward_deepgemm_contiguous and forward_deepgemm_masked. Finalize deep_gemm backend.

Modifications

Accuracy Tests

Benchmarking and Profiling

Server:

python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3-0324 --trust-remote-code --tp 8 --enable-dp-attention --dp 8 --moe-dense-tp-size 1 --enable-dp-lm-head --moe-a2a-backend deepep --enable-two-batch-overlap --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic --eplb-algorithm deepseek --cuda-graph-bs 256 --max-running-requests 2048 --disable-radix-cache --device cuda

Benchmark:

python3 -m sglang.bench_serving --backend sglang --dataset-name random --num-prompt 512 --random-input 1000 --random-output 1000 --random-range-ratio 1 --host 127.0.0.1 --port 30000 --max-concurrency 128

Token throughputs are both around 2120.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[ch-wan]

/gemini review

[gemini-code-assist]

Code Review

This pull request refactors the Mixture-of-Experts (MoE) layer by reorganizing the deepgemm runner logic. The core change is deprecating forward_deepgemm_contiguous and forward_deepgemm_masked from DeepEPMoE and moving their functionality into a more modular, runner-based architecture within deep_gemm.py. This is a solid architectural improvement that enhances code clarity and maintainability. The introduction of CombineInput wrappers and renaming of DispatchOutput classes also contribute to better code readability. I've identified a couple of potential memory leaks due to missing tensor disposals and a minor point about using assert False for deprecation warnings. Overall, this is a great step towards a cleaner MoE implementation.