[Feature] pipeline parallel with mixed chunk

[MichoChan]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

i test, find it seems like that pipeline parallel with mixed chunk worked well?
so why assert this:
if self.pp_size > 1: assert ( self.disable_overlap_schedule and self.speculative_algorithm is None and not self.enable_mixed_chunk ), "Pipeline parallelism is not compatible with overlap schedule, speculative decoding, mixed chunked prefill."

Related resources

No response

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

[xesdiny]

Pipeline Parallelism with Mixed Chunked Prefill: Reevaluation Request

Summary

This issue requests a reevaluation of the assertion that prevents Pipeline Parallelism (PP) from being used with Mixed Chunked Prefill. Evidence suggests that this restriction may be outdated following recent PP refactoring and improvements.

Background

Current Restriction

In python/sglang/srt/server_args.py:4512-4517, the following assertion prevents PP from being used with several features:

# FIXME pp constraints
if self.pp_size > 1:
    logger.warning(f"Turn off overlap scheule for pipeline parallelism.")
    self.disable_overlap_schedule = True
    assert (
        self.disable_overlap_schedule
        and self.speculative_algorithm is None
        and not self.enable_mixed_chunk
    ), "Pipeline parallelism is not compatible with overlap schedule, speculative decoding, mixed chunked prefill."

This restriction was introduced in commit 11383cec3 (PR #5724, April 30, 2025) when PP was first added. Notably, the code includes a # FIXME pp constraints comment, suggesting these limitations were intended to be temporary.

Related Issue

Issue #10589 raised the same question, with the author reporting that PP + mixed chunk "worked well" in their testing. However, this issue was automatically closed due to inactivity without receiving any substantive technical response from maintainers.

Analysis of Each Restriction

1. Overlap Schedule - Legitimately Incompatible ✓

The restriction on overlap schedule is technically justified. The scheduler's event loop selection in python/sglang/srt/managers/scheduler.py:2938-2945 shows mutually exclusive execution paths:

if scheduler.enable_pdmux:
    scheduler.event_loop_pdmux()
elif server_args.pp_size > 1:
    scheduler.event_loop_pp()           # PP-specific event loop
elif scheduler.enable_overlap:
    scheduler.event_loop_overlap()      # Overlap-specific event loop
else:
    scheduler.event_loop_normal()

PP uses a completely different scheduling loop implementation (event_loop_pp) that is fundamentally incompatible with the overlap scheduling mechanism.

2. Speculative Decoding - Likely Incompatible ✓

The restriction on speculative decoding appears justified due to implementation complexity. Coordinating draft model execution across multiple PP stages would require significant architectural changes.

3. Mixed Chunked Prefill - Restriction Appears Outdated ⚠️

This is where the evidence becomes contradictory. Multiple code paths suggest PP + mixed chunk should work together:

Evidence Supporting Compatibility:

1. Mixed chunk initialization doesn't check PP constraints:

In python/sglang/srt/managers/scheduler.py:752-755:

self.is_mixed_chunk = (
    self.chunked_prefill_size is not None
    and self.server_args.enable_mixed_chunk  # No pp_size check here
)

2. Dynamic chunking explicitly designed for PP:

In python/sglang/srt/managers/scheduler.py:758-760:

# Init the dynamic chunking predictor for PP
self.enable_dynamic_chunking = (
    self.server_args.enable_dynamic_chunking and self.pp_size > 1
)

This shows that chunking strategies are being actively developed for PP workloads.

3. PP event loop calls mixed chunk logic:

The event_loop_pp() in python/sglang/srt/managers/scheduler_pp_mixin.py:84-85 calls:

with torch.profiler.record_function("get_next_batch_to_run"):
    self.mbs[mb_id] = self.get_next_batch_to_run()

And get_next_batch_to_run() includes the mixed chunk implementation in scheduler.py:2055-2064:

# Mixed-style chunked prefill
if (
    self.is_mixed_chunk
    and not self.running_batch.is_empty()
    and not (new_batch.return_logprob or self.running_batch.return_logprob)
):
    # TODO (lianmin): support return_logprob + mixed chunked prefill
    self.running_batch.filter_batch(v1_spec_info_filtered=True)
    if not self.running_batch.is_empty():
        self.running_batch.prepare_for_decode()
        new_batch.mix_with_running(self.running_batch)
        new_batch.decoding_reqs = self.running_batch.reqs
    self.running_batch = ScheduleBatch(
        reqs=[], batch_is_full=self.running_batch.batch_is_full
    )

This code path would be executed when PP is active, suggesting the implementation supports this combination.

4. Recent PP improvements related to chunking:

• PR [PP] Refactor PP to async mode #11852 (December 12, 2025): Major refactor of PP to async mode, which significantly improved PP's architecture
• PR [PP] Fix dynamic chunking strategy for PP #15372 (December 18, 2025): "Fix dynamic chunking strategy for PP" - Recent active development of chunking for PP
• PR [Ascend] Support enable-mixed-chunk in non-MLA scenarios #12491: "Support enable-mixed-chunk in non-MLA scenarios" - Shows continued development of mixed chunk support

Inconsistency Summary

There's a clear contradiction:

1. server_args.py forbids PP + mixed chunk with an assertion that has not been updated since PP was first introduced (April 2025)

2. Scheduler implementation includes mixed chunk logic that will execute in PP's event loop

3. Active development continues on chunking strategies specifically for PP (dynamic chunking, recent fixes)

4. User testing ([Feature] pipeline parallel with mixed chunk #10589) reported that the combination works in practice

Request

Given the evidence above, I request that the SGLang team:

1. Clarify the technical rationale for prohibiting mixed chunk with PP, if one still exists

2. Test the combination to verify whether it actually works or causes issues

3. Update the assertion to either:

   • Remove the enable_mixed_chunk restriction if it works
   • Add detailed comments explaining why it doesn't work, if there are actual technical blockers
   • Update the # FIXME comment with a timeline or requirements for lifting the restriction

4. Consider the precedent set by enable_dynamic_chunking, which successfully combines PP with chunking strategies

Potential Impact

If this restriction is indeed outdated, removing it would:

• Allow users to benefit from both PP and mixed chunk prefill simultaneously
• Improve throughput in scenarios where both features would be beneficial
• Reduce confusion about feature compatibility
• Align the codebase constraints with the actual implementation capabilities

References

• Original issue: [Feature] pipeline parallel with mixed chunk #10589 (closed as inactive without resolution)
• PP introduction: PR [PP] Add pipeline parallelism #5724, commit 11383cec3 (April 30, 2025)
• PP async refactor: PR [PP] Refactor PP to async mode #11852 (December 12, 2025)
• PP dynamic chunking fix: PR [PP] Fix dynamic chunking strategy for PP #15372 (December 18, 2025)
• Code references:
  • Assertion: python/sglang/srt/server_args.py:4512-4517
  • Mixed chunk check: python/sglang/srt/managers/scheduler.py:752-755
  • Dynamic chunking: python/sglang/srt/managers/scheduler.py:758-760
  • Mixed chunk implementation: python/sglang/srt/managers/scheduler.py:2055-2064
  • PP event loop: python/sglang/srt/managers/scheduler_pp_mixin.py:42-149

---

Note: I am willing to test this combination and provide detailed results if the maintainers would find that helpful. The # FIXME comment suggests this was always intended to be revisited.