add all lycoris models

[ljleb]

This issue tracks support for lycoris and popular peft models in the latest version of sd-mecha.

• boft
• diag-oft
• dylora
• full
• glora
• ia3
• locon
• kohya lora
• loha
• lokr

[Enferlain]

This is about the mixed lora I've been mentioning:

Hit a mixed LyCORIS adapter that looks like LoKr + Norm, with a small plain LoRA/LoCon subset too, and current sd-mecha appears to only recognize the existing LoRA/LoKr-shaped parts while missing the norm part.

Example file:
Big-test-RF.safetensors (talked about this on the server a few times)

Observed locally with 1.0.14 -1.1.3:

• total keys: 576
• inferred candidate config: sdxl-kohya_kohya_lora
• matched keys: 66

Key breakdown:

• 255 keys ending in w_norm
• 255 keys ending in b_norm
• 19 lokr_w1
• 19 lokr_w2
• 22 alpha
• 6 LoRA/LoCon weights:
  • lora_unet_out_2.lora_down.weight
  • lora_unet_out_2.lora_up.weight
  • lora_unet_time_embed_0.lora_down.weight
  • lora_unet_time_embed_0.lora_up.weight
  • lora_unet_time_embed_2.lora_down.weight
  • lora_unet_time_embed_2.lora_up.weight

This seems consistent with a mixed LyCORIS adapter where:

• lokr_* is already supported
• the 6 lora_* weights are normal LoRA/LoCon-style keys
• w_norm / b_norm come from LyCORIS norm / NormModule

Since norm is not on the support tracker list above, mixed Thingy + Norm state dicts would only partially be recognized because the norm part is ignored.

Guess for the fix from gpt:

• add LyCORIS norm support (w_norm, b_norm) lycoris/modules/norms.py
• allow algorithm-specific base-key eligibility, since norm needs 1D norm weights/biases while LoRA/LoKr stay matrix-oriented
• then mixed LyCORIS configs/conversions should be able to union LoKr + LoRA + Norm relations per key naturally

[ljleb]

Ah this is super useful info, thanks. I'll prioritize this, will be included in the next version.

[ljleb]

ah we can't extract delta norms without knowing what the base model is. we need a new interface merge method sd_mecha.apply_lycoris(base, lyco) for this to work.

[ljleb]

In fact, the w_norm and b_norm keys are deltas: https://github.com/KohakuBlueleaf/LyCORIS/blob/72d7e5af6c4b133153e2c41b5e5ca2bd4327a4a4/lycoris/modules/norms.py#L92. So there is no need to introduce apply_lycoris() in the next update. I've however already made significant progress on implementing it, so I'll keep the changes regardless.