How to load multiple LORAs while adjusting the weight of different LORAs

[GanymedeNil]

I saw that #263 supports multiple LORAs, but it looks like it only supports switching multiple LORAs, not multiple LORA loading at the same time and supports adjusting the corresponding weights, if I want to achieve similar results is there a supported method currently? 🤔

[pacman100]

not multiple LORA loading at the same time and supports adjusting the corresponding weights

Could you tell us why that would be useful to a broader community and provide example applications?

[pacman100]

Currently, you can load multiple adapters and merge into base model as following, here all are weighted equally:

model = AutoModel... # base model # set `load_in_8bit` to `False`
for peft_model_id in peft_model_ids:
    model = PeftModel.from_pretrained(model, peft_model_id)
    model = model.merge_and_unload()

[GanymedeNil]

not multiple LORA loading at the same time and supports adjusting the corresponding weights

Could you tell us why that would be useful to a broader community and provide example applications?

I'm not sure if this is a common requirement, but I can share my example: I am currently working on an AI service that mainly trains on daily conversations among professionals, as well as theoretical knowledge related to professional fields. The ultimate goal is to provide answers that are closer to the style of daily conversations between people, rather than producing rigid theoretical knowledge. Therefore, I hope to be able to load multiple LORAs at the same time and fuse them based on certain weights when asking questions, such as in this case, the LORAs for daily conversation and professional knowledge. It's similar to combining different styles of images in text-to-image generation.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[Dessix]

Still relevant.

Stalebots eat all of the useful feature requests.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

[Dessix]

Still relevant, yet again.

[younesbelkada]

This is an interesting feature, added it in our internal roadmap.
EDIT: seems to be already supported, refer to the comments below

[pacman100]

Hello, this is already supported. See the example with respect to Diffusers Dreambooth using PEFT showing how to combine multiple adapters as a weighted combination.

https://github.com/huggingface/peft/blob/main/examples/lora_dreambooth/lora_dreambooth_inference.ipynb

Just apply the same to NLP tasks

[pacman100]

This is actually solved

[GanymedeNil]

Glad to feel that a solution has been found. I will close this issue.