Make refiner switchover based on model timesteps instead of sampling steps#14978
Merged
AUTOMATIC1111 merged 4 commits intoAUTOMATIC1111:devfrom Mar 2, 2024
Merged
Make refiner switchover based on model timesteps instead of sampling steps#14978AUTOMATIC1111 merged 4 commits intoAUTOMATIC1111:devfrom
AUTOMATIC1111 merged 4 commits intoAUTOMATIC1111:devfrom
Conversation
4 tasks
Contributor
Author
|
I will note that there are two remaining cases that are handled poorly by the refiner:
There is no proper way to handle these cases without having the refiner switch on and off multiple times during sampling. Neither particularly worked well with refiners to begin with, and Restart is the only one that could be argued to be a regression. I can look into ways to get Restart to only switch to the refiner on the last "restart", which would have Restart working as well as it could have before, but I don't think that robust handling of both of these cases would have benefits that justify the extra complexity required. |
AUTOMATIC1111
approved these changes
Mar 2, 2024
AUTOMATIC1111
added a commit
that referenced
this pull request
Mar 2, 2024
ruchej
pushed a commit
to ruchej/stable-diffusion-webui
that referenced
this pull request
Sep 30, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
edit: converted to draft while I troubleshoot an off-by-one errorThere is a bug where model alphas_cumprod changes (compatibility casting option or zero terminal snr) are reverted during the timestep the refiner is applied. This will have to be fixed separately.Screenshots/videos:
Examples of old behavior in txt2img:

The refiner model used here was trained for the last 200 timesteps. The Karras schedule type, especially on zero snr, drastically changes the model timesteps called during this 50 step sampling process, which results in the refiner being switched to too early on what is actually the correct setting on the default noise schedule. The effectively correct setting for Karras samplers is 0.88 for this refiner under the old configuration.
Now for the fixed version:

With this fix, the behavior of the refiner is consistent with the same settings across different schedules, and it no longer triggers too early. 0.8 is reliably a correct setting.
Examples of old behavior in img2img/inpainting (inpainting mask is over the head, adding a hat, 0.75 denoising strength):


This one is more complicated, and the differences are subtle. The effectively correct settings for the normal schedule is to switch over at 0.75, and for Karras it is correct to switch over at 0.85. Using the expected setting of 0.8 therefore is too late for normal schedules and too early for Karras ones. As denoising strength gets lower, the problem becomes more severe.
This grid shows the behavior after the fix. Switch at 0.8 is now correct for both.
Checklist: