[Bug]: Switching to another model and then back leads to different image (SDXL)

[djdookie]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

If I generate an image with sdxl_base_1.0_0.9vae with one self trained LoRa (with strength 0.8),
then switch the model to another custom SDXL1.0 model, generate another image with the exact same prompt,
and then switch back to the first model and generate an image with the exact same parameters again, it looks different/broken.

This only happens if I use a LoRa.

When I completely restart the webUI I can generate the correct first image again.

So for me it seems that something is not unloaded/replaced in memory correctly.

Edit: It looks like the LoRa is applied with strength 1.0 (instead of 0.8) after switching model and switching back.
And it is always applied even if I set it to strength 0.0 or remove it from prompt!

Steps to reproduce the problem

1. Go to ....
2. Press ....
3. ...

What should have happened?

the last image should be the same as the first

Version or Commit where the problem happens

v1.5.1, also tried current dev branch build, same issue

What Python version are you running on ?

Python 3.11.x (above, no supported yet)

What platforms do you use to access the UI ?

Windows

What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

Cross attention optimization

sdp

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

no

List of extensions

openpose-editor, sd-webui-controlnet, sd-webui-openpose-editor, sd-webui-refiner, sd-webui-roop-nsfw, ultimate-upscale-for-automatic1111

Console logs

nothing relevant

Additional information

No response

[w-e-w]

cannot reproduce issue sdxl 1.0 lora 0.8 switch back to sdxl 1.0 lora 0.8 are identical

sdxl 1.0 lora 1.0	sdxl 1.0 lora 0.8	other xl model lora 0.8	switch back to sdxl 1.0 lora 0.8

2023-08-17.13_15_10_906.chrome.mp4

[djdookie]

Thanks for testing.
Could this be associated with my python 3.11.x version?
Or the selftrained lora itself (used kohya_ss repo)? Or the model I switch to (dynavisionXL)?

[w-e-w]

I would try to disable third party extensions and test again
I think say that the chance of this being caused by a model should be low (assuming that the models is trained correctly)
if you want you can upload your model / give me links so I can test it with your model see if it is the cause
but I think most likely it's either caused by some extension or you accidentally made a human error

[djdookie]

Yeah I will test disabling extensions. I can 100% reproduce that issue and did that multiple times, even after restarting and reloading everything. You can find dynaVisionXL on civitai. I will test with another lora as soon as I find the time.

[w-e-w]

when you have time also test with a completely new install
with every setting aside from the models as default

[TomTomGit86]

I'm having this same issue too! Really frustrating when the image you want to reproduce or go back to and tweak the prompt a little to slight change it is now yielding completely different results.
I have had similar thoughts such that the problem is like it holding on the the previous model data when switching between models.
Also the Lora strength seems to have changed or disappeared.
Seems funny that I have used the model DynavisionXL also but wasn't the model Im currently using and experiencing the issues.

I'm now using model:https://civitai.com/models/119279
And https://civitai.com/models/123307/sdvn7-realartxl
And
Then I also created a merge of these to as I thought the desirable images had came about as an accidental bugged merging of the two models and I'm getting some good images with the merged model but then still having the issue where I go back to a image/seed/config to reproduce only to find a completely different output.

I'm using a Lora:
https://civitai.com/models/129533?modelVersionId=141997

I tend to create a txt2img then having found a good starting image I use loopback wave script in img2img to create a set of image frames to animate with.
Sometimes the set of images aren't as id wanted so I go back to the txt2img phase to create a new starting image, and tweak some negative to eliminate some of the undesirable resulting img2img frames only to have out of line results. So I revert back to exactly the same promt seed etc to check I'm not going mad and there it is two different images created with the same promt seed and identical settings

Hope we can get to the bottom of why this maybe happening.

I'm running rtx2070 maxq (8gbVram)
And 16gb sys ram.
I bought upgrade of 64gb sys ram which should arrive today as I noticed I'm allways at the limit.

And I've never been able to load the base sdxl model successfully it seems to out of memory error/crash.

I can only seem to load custom sdxl models.

I will do some more testing playing around and post any more updates here.

Likewise if you need any more info data from me to help fix this issue I'm happy to help.

[w-e-w]

make sure that what you're seeing is not caused by SDP
as SDP is not deterministic that results will not be the same
but generally speaking the difference to a human size is barely noticeable but it's not Pixel Perfect
however under certain conditions the variance can be amplified

so if what you're saying is SDP not deterministic then it's not a bug

[TomTomGit86]

make sure that what you're seeing is not caused by SDP as SDP is not deterministic that results will not be the same but generally speaking the difference to a human size is barely noticeable but it's not Pixel Perfect however under certain conditions the variance can be amplified

so if what you're saying is SDP not deterministic then it's not a bug

--opt-sdp-attention

I have started using the above argument as suggested for performance/time to generate images

But the difference in the images are greatly noticeable
Like the Lora is missing or changed weight I will upload some examples images with exactly the same generation data when I get back home so you can see what you think yourself, I may upload model and images to Google drive so maybe others could test gen the same and see what outputs they get I plan on trying a fresh install and rerunning the generations in question again to see what results are.

[djdookie]

I don't think this is a SDP problem, since this leads to only minor changes in the generated image.
After restarting the webUI I get back to mostly the same picture.

This is what I am talking about:

1. Initial image with SDXL base and my lora:0.8:	2. Switch to dynaVisionXL and my lora:0.8:	3. Switch back to SDXL base and my lora:0.8:

I tried some more things now:

• reverted back to recommended python version 3.6.10.
• disabled all extra extensions.
• reinstalled webui
• use sdxl_vae_sp16_fix

Problem still persists, 100% reproducible with my own LoRa!

But I tested some other LoRas from civitai and wasn't able to reproduce the issue with other LoRas yet.
What could be wrong with my self-trained LoRa to trigger that problem?

I tried the same in vladmandics webui fork and was able to get black images this way (in the VAE decode phase), even with the sdxl_vae_sp16_fix. (I trained my LoRa with no half vae parameter to prevent NaNs)
There I triggered this error, possibly this could help to narrow down the issue?

It's possible I am doing something wrong in my LoRa training, but I don't know what this could be to trigger such issues in the webUI.

[TomTomGit86]

hi again here are the two image examples using the exact same promt/model/settings etc and as you can see rather more diffrent than i would perhaps expect from what i now know can be the case when implementing argumenta like xformers.
im thinking more so now this is perhaps an issue with the lora its self. i cant say 100% but i dont think furing the generations ibswitched models at any point i did however change the lora wieght and possibly restarted auto1111.
if ibrember correctly it was only after resarting auto1111 did i yield the much diffrent image, where as prior i could replicate it .

[TomTomGit86]

@djdookie
You could see if you can recreate the problem with the Lora I'm using that might then point to the problem being with the Lora's
https://civitai.com/models/129533?modelVersionId=141997
V1. 0 of the Lora
I have just seen there's an update to the Lora I'm using also now maybe because of the user noticed the same issue we've had possibly leave a comment on civ and see if the author has any tips/help for you making your Lora's

[w-e-w]

damn I see it