PEFT Integration for Text Encoder to handle multiple alphas/ranks, disable/enable adapters and support for multiple adapters#5147
Merged
patrickvonplaten merged 63 commits intomainfrom Sep 27, 2023
Conversation
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
sayakpaul
reviewed
Sep 25, 2023
sayakpaul
reviewed
Sep 25, 2023
sayakpaul
reviewed
Sep 25, 2023
sayakpaul
reviewed
Sep 25, 2023
Member
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Agree with Patrick's and Benjamin's comments. Mine are mostly clarification questions and nits.
Co-Authored-By: Benjamin Bossan <BenjaminBossan@users.noreply.github.com> Co-Authored-By: Patrick von Platen <patrick.v.platen@gmail.com>
sayakpaul
reviewed
Sep 27, 2023
| lora_scale=self.lora_scale, | ||
| low_cpu_mem_usage=low_cpu_mem_usage, | ||
| _pipeline=self, | ||
| adapter_name=adapter_name, |
Member
There was a problem hiding this comment.
Let's keep the private variable at the end.
sayakpaul
reviewed
Sep 27, 2023
sayakpaul
approved these changes
Sep 27, 2023
Contributor
|
For the tests, I am happy to add them here: #5151 to complete the full testing suite See my comment: #5151 (comment) |
|
The documentation is not available anymore as the PR was closed or merged. |
Contributor
Ok for me to add tests in a later PR |
Contributor
|
Very nice job @pacman100 - think the design is very nice & also agree that higher level |
patrickvonplaten
approved these changes
Sep 27, 2023
Contributor
patrickvonplaten
left a comment
patrickvonplaten
left a comment
There was a problem hiding this comment.
Ok for me to merge once CI is green (ok if LoRA peft tests are still failing)
yoonseokjin
pushed a commit
to yoonseokjin/diffusers
that referenced
this pull request
Dec 25, 2023
…sable/enable adapters and support for multiple adapters (huggingface#5147) * more fixes * up * up * style * add in setup * oops * more changes * v1 rzfactor CI * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * few todos * protect torch import * style * fix fuse text encoder * Update src/diffusers/loaders.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * replace with `recurse_replace_peft_layers` * keep old modules for BC * adjustments on `adjust_lora_scale_text_encoder` * nit * move tests * add conversion utils * remove unneeded methods * use class method instead * oops * use `base_version` * fix examples * fix CI * fix weird error with python 3.8 * fix * better fix * style * Apply suggestions from code review Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * add comment * Apply suggestions from code review Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * conv2d support for recurse remove * added docstrings * more docstring * add deprecate * revert * try to fix merge conflicts * peft integration features for text encoder 1. support multiple rank/alpha values 2. support multiple active adapters 3. support disabling and enabling adapters * fix bug * fix code quality * Apply suggestions from code review Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> * fix bugs * Apply suggestions from code review Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> * address comments Co-Authored-By: Benjamin Bossan <BenjaminBossan@users.noreply.github.com> Co-Authored-By: Patrick von Platen <patrick.v.platen@gmail.com> * fix code quality * address comments * address comments * Apply suggestions from code review * find and replace --------- Co-authored-by: younesbelkada <younesbelkada@gmail.com> Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
AmericanPresidentJimmyCarter
pushed a commit
to AmericanPresidentJimmyCarter/diffusers
that referenced
this pull request
Apr 26, 2024
…sable/enable adapters and support for multiple adapters (huggingface#5147) * more fixes * up * up * style * add in setup * oops * more changes * v1 rzfactor CI * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * few todos * protect torch import * style * fix fuse text encoder * Update src/diffusers/loaders.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * replace with `recurse_replace_peft_layers` * keep old modules for BC * adjustments on `adjust_lora_scale_text_encoder` * nit * move tests * add conversion utils * remove unneeded methods * use class method instead * oops * use `base_version` * fix examples * fix CI * fix weird error with python 3.8 * fix * better fix * style * Apply suggestions from code review Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * add comment * Apply suggestions from code review Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * conv2d support for recurse remove * added docstrings * more docstring * add deprecate * revert * try to fix merge conflicts * peft integration features for text encoder 1. support multiple rank/alpha values 2. support multiple active adapters 3. support disabling and enabling adapters * fix bug * fix code quality * Apply suggestions from code review Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> * fix bugs * Apply suggestions from code review Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> * address comments Co-Authored-By: Benjamin Bossan <BenjaminBossan@users.noreply.github.com> Co-Authored-By: Patrick von Platen <patrick.v.platen@gmail.com> * fix code quality * address comments * address comments * Apply suggestions from code review * find and replace --------- Co-authored-by: younesbelkada <younesbelkada@gmail.com> Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?