[Feature] Mooncake CPP (Chunked Pipeline Parallelism)

[kingzevin]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Hi team,

I found that the sequence parallelism (Meta CP based on https://www.arxiv.org/pdf/2411.01783) is in the roadmap (#4042). For long-context inference, I'm wondering do you have plans to support CPP (chunked pipeline parallelism) from Mooncake or SPP (sequence pipeline parallelism) from Mnemosyne?
If there are no plans, could you please leave some comments about your thoughts on the above CPP/SPP technology for long-context prefilling, compared with Context Parallelism? Really appreciate your attention~

Related resources

No response

[FlamingoPg]

really need this

[FlamingoPg]

Need pp firstly @zhyncs

[Ying1123]

@yinfan98 Supported in

• [PP] Add pipeline parallelism #5724

[kingzevin]

This is amazing — thank you so much for implementing this feature so quickly!

@yinfan98 Supported in

• [PP] Add pipeline parallelism #5724

[jifa513]

Need pp firstly @zhyncs

How effective is CPP?

[Zhangmj0621]

@kingzevin @Ying1123 Why close this feature? It seems that vllm only support pipeline parallelism not chunked pp.

[kingzevin]

@kingzevin @Ying1123 Why close this feature? It seems that vllm only support pipeline parallelism not chunked pp.

Already supported in SGLang.

[Zhangmj0621]

@kingzevin @Ying1123 Why close this feature? It seems that vllm only support pipeline parallelism not chunked pp.

Already supported in SGLang.

Sorry I mistake vllm and sglang. But I wonder In the pr above, it seems that only pipeline parallelism is supported in sglang, not chunked pipeline parallelism of pd disaggregation. Could you show me some pr related with chunked pipeline parallelism? I would highly appreciate any feedbacks from you.

[kingzevin]

CPP is not supported in PD disaggregation but only in PD colocation mode. To enable it, you can just look at the arguments about chunked-prefill-size and pp.

[Zhangmj0621]

CPP is not supported in PD disaggregation but only in PD colocation mode. To enable it, you can just look at the arguments about chunked-prefill-size and pp.

In fact, chunked prefill is mainly used for PD colocation mode. However, CPP is presented for PD disaggregation in mooncake to address long context. Sglang now support pp(used for request-level parallelism). I wonder if the parallelsim of long contexts(divided to several chunks) hasn't been implemented by sglang.

[kingzevin]

I agree with you that CPP is important for PD disaggregation.
If I understand it correctly, the current chunked-prefill + PP can already do the task of 'parallelism + divided to several chunks' you've mentioned. I think that IS CPP.
What's the difference between CPP in PD colocation vs in disaggregation?

[Zhangmj0621]

I agree with you that CPP is important for PD disaggregation. If I understand it correctly, the current chunked-prefill + PP can already do the task of 'parallelism + divided to several chunks' you've mentioned. I think that IS CPP. What's the difference between CPP in PD colocation vs in disaggregation?

What bothers me is that the support of chunked prefill and pipeline parallelism don't equal to cpp support. CPP means for one long prompt, when its first chunk is calculated in the second p state, next chunk can start in the first p stage, it is chunk-level. However, pp means that, when one request is calculated in the second p state, next request can start in the first p stage, it is request-level. For chunked prefill itself, it needs the second chunk to start computing after all the first chunk complete computing. So I wonder if the support of chunked prefill and pp is equal to CPP?