[wip] [core] introduce pipeline-specific mixins to hold common methods

[sayakpaul]

What does this PR do?

As discussed internally, this PR introduces pipeline-specific Mixin classes that hold common methods shared across different task-specific pipelines. The actual pipelines (such as StableDiffusionPipeline, QwenImagePipeline, QwenImageImg2ImgPipeline, etc.) would then subclass from these mixins.

Examples of such common methods vary from pipeline to pipeline, but some examples include encode_prompt(), properties such as guidance_scale, etc.

Additionally, we have a couple of methods like retrieve_latents() and retrieve_timesteps() originated in stable_diffusion and we copied them over to other pipelines that use these methods. Same for calculate_shift that originated in flux. This PR also considers this situation and follows a reasonable approach to treat it (more below).

I know we also discussed moving the specific LoraLoaderMixins to their respective pipeline modules. The PR only does that for Stable Diffusion to gather feedback. Please keep on reading.

Note

This PR introduces Mixin classes for four popular pipelines: Flux, Qwen, SDXL, and SD. Once the PR is merged, we could work with the community to do this for other influential pipelines.

---

Guiding principles

A sample structure for src/diffusers/pipelines/flux:

.
├── __init__.py
├── modeling_flux.py
├── pipeline_flux_control_img2img.py
├── pipeline_flux_control_inpaint.py
├── pipeline_flux_control.py
├── pipeline_flux_controlnet_image_to_image.py
├── pipeline_flux_controlnet_inpainting.py
├── pipeline_flux_controlnet.py
├── pipeline_flux_fill.py
├── pipeline_flux_img2img.py
├── pipeline_flux_inpaint.py
├── pipeline_flux_kontext_inpaint.py
├── pipeline_flux_kontext.py
├── pipeline_flux_prior_redux.py
├── pipeline_flux_utils.py <------ holds the Mixin
├── lora_utils.py <------ holds the `FluxLoraLoaderMixin`
├── pipeline_flux.py
└── pipeline_output.py

pipeline_flux_utils.py here has the Mixin class, holding the common methods. It additionally has the calculate_shift() method, which is imported by the other pipelines under src/diffusers/pipelines/flux. Those pipelines won't have to maintain # Copied from ... versions of the calculate_shift() method anymore. They can just import like from .pipeline_flux_utils import calculate_shift. This is how we reduce LOCs, further on an intra-pipeline basis.

However, if a new pipeline (other than Flux), wants to leverage calculate_shift(), it should still follow # Copied from ... versions of the calculate_shift() instead of a direct import. This way, we can let individual pipeline-level methods evolve independently without introducing complex dependency patterns. For example, pipeline_flux_utils.py uses the # Copied from ... versions of methods like retrieve_latents().

LoRA

For LoRA, I have limited the changes to only Flux. Once we settle on that, I can propagate that change for other pipelines tackled in this PR.

---

Notes

• In the Qwen mixin, kept encode_prompt() varying while unifying the _get_qwen_prompt_embeds() method in QwenImageMixin. Comments are in line.
• There are a few methods that are shared by 2/3 pipelines such as prepare_image() is shared by QwenImageControlNetInpaintPipeline and QwenImageControlNetPipeline but not others. We can choose to move them to pipeline_qwen_utils.py but I am not strongly opinionated on this. The same applies to methods like prepare_mask_latents(). My preference here is to keep them as is for now.

We can take similar principles for Flux, too. Flux Control and ControlNet pipelines have a prepare_image() method. So, we could have something like FluxControlMixin(FluxMixin) and include prepare_image() there. But I felt like it was getting complex so, decided to skip.

---

Additional questions

We have

diffusers/src/diffusers/pipelines/pipeline_utils.py

Line 2175 in 6bf668c

class StableDiffusionMixin:

This PR also has a similar Mixin class but without the methods from the above Mixin. To avoid confusion, I have named it SDMixin.

WDYT about collating the two classes (StableDiffusionMixin and SDMixin (introduced in this PR)) into one?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.