[Bug]: Bug in AUTOMATIC1111 when running ControlNet and Inpaint in "Only masked" and "Mask blur" > 0

[CrazyMaxTM]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What happened?

When working with Inpaint in the "Only masked" mode and "Mask blur" greater than zero, ControlNet returns an enlarged image (by the amount of Mask blur), as a result of which the area under the mask increases:

These settings were used:

These settings gave the same result.:

my software version Windows 10

I use a 12Gb RTX3060 graphics card, 16Gb RAM

This is definitely a mistake, since mask blur should lead to an increase in the width of the mixing zone between the embedded image (drawn in Inpaint) and the source, and not to an increase in the image inside the mask. I repeat, everything worked correctly before and I have a backup copy of AUTOMATIC1111 that has not been updated for several months and everything works correctly there!!! And it seems to me that the problem is in AUTOMATIC 1111, since updating ControlNet to the latest version in my backup did not lead to an error, but updating AUTOMATIC1111 led to its occurrence.

Steps to reproduce the problem

1. Upgrade AUTOMATIC 1111 to version 1.5.1

2. Go to Image To Image -> Inpaint put your picture in the Inpaint window and draw a mask. Set Mask Blur > 0 (for example 16).

3. Enable the "Only masked" option. To clearly see the result, set Denoising strength large enough (for example = 1)

4. Turn on ControlNet and put the same picture there. Use the settings as in the picture (I used a tile and a soft edge, but you can use one thing the result will be the same

5. Click Generate and get the result:

What should have happened?

This is definitely a mistake, since mask blur should lead to an increase in the width of the mixing zone between the embedded image (drawn in Inpaint) and the source, and not to an increase in the image inside the mask.

Here is another example with another picture:

If Mask blur=0 then everything is correct:

Version or Commit where the problem happens

v1.5.1

What Python version are you running on ?

Python 3.10.x

What platforms do you use to access the UI ?

Windows

What device are you running WebUI on?

Nvidia GPUs (RTX 20 above)

Cross attention optimization

sdp-no-mem

What browsers do you use to access the UI ?

No response

Command Line Arguments

@echo off
set PYTHON=
set GIT=
set VENV_DIR=
set CUDA_MODULE_LOADING=LAZY 
set SAFETENSORS_FAST_GPU=1
set PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.9,max_split_size_mb:512
set COMMANDLINE_ARGS=--opt-sdp-no-mem-attention --upcast-sampling --autolaunch --ckpt-dir "G:\StableDiffuzion-Models\Stable-diffusion" --lora-dir "G:\StableDiffuzion-Models\Lora" --lyco-dir "G:\StableDiffuzion-Models\LyCORIS" --embeddings-dir "G:\StableDiffuzion-Models\Embeddings" --hypernetwork-dir "G:\StableDiffuzion-Models\Hypernetworks" --esrgan-models-path "G:\StableDiffuzion-Models\ESRGAN" --vae-dir "G:\StableDiffuzion-Models\VAE"
call webui.bat

But I also checked for this bug in the new clean install of AUTOMATIC 1111 + ControlNet only. There were such command line arguments:

@echo off
set PYTHON=
set GIT=
set VENV_DIR=
set COMMANDLINE_ARGS=--opt-sdp-attention --autolaunch
call webui.bat

List of extensions

Вut I also checked for this bug in the new clean install of AUTOMATIC 1111+ ControlNet only

Console logs

venv "E:\Stable-Diffusion-webui-update\venv\Scripts\Python.exe"
Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec  6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)]
Version: v1.5.1
Commit hash: 68f336bd994bed5442ad95bad6b6ad5564a5409a


Fetching updates for midas...
Checking out commit for midas with hash: 1645b7e...




Installing pyqt5 requirement for depthmap script


Launching Web UI with arguments: --opt-sdp-no-mem-attention --upcast-sampling --autolaunch --ckpt-dir G:\StableDiffuzion-Models\Stable-diffusion --lora-dir G:\StableDiffuzion-Models\Lora --lyco-dir G:\StableDiffuzion-Models\LyCORIS --embeddings-dir G:\StableDiffuzion-Models\Embeddings --hypernetwork-dir G:\StableDiffuzion-Models\Hypernetworks --esrgan-models-path G:\StableDiffuzion-Models\ESRGAN --vae-dir G:\StableDiffuzion-Models\VAE
No module 'xformers'. Proceeding without it.
Civitai Helper: Get Custom Model Folder
Civitai Helper: Load setting from: E:\Stable-Diffusion-webui-update\extensions\Stable-Diffusion-Webui-Civitai-Helper\setting.json
2023-08-09 12:30:38,460 - ControlNet - INFO - ControlNet v1.1.237
ControlNet preprocessor location: E:\Stable-Diffusion-webui-update\extensions\sd-webui-controlnet\annotator\downloads
2023-08-09 12:30:38,534 - ControlNet - INFO - ControlNet v1.1.237
sd-webui-prompt-all-in-one background API service started successfully.
Loading weights [4d313d1209] from G:\StableDiffuzion-Models\Stable-diffusion\R\aZovyaPhotoreal_v1Ultra.safetensors
Creating model from config: E:\Stable-Diffusion-webui-update\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: G:\StableDiffuzion-Models\VAE\vae-ft-mse-840000-ema-pruned.ckpt
Applying attention optimization: sdp-no-mem... done.
No Image data blocks found.
Model loaded in 5.0s (load weights from disk: 0.4s, create model: 0.4s, apply weights to model: 1.4s, apply half(): 0.7s, load VAE: 0.2s, move model to device: 1.1s, load textual inversion embeddings: 0.5s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 26.8s (launcher: 11.3s, import torch: 3.0s, import gradio: 0.9s, setup paths: 0.5s, other imports: 0.7s, setup codeformer: 0.2s, list SD models: 0.1s, load scripts: 2.7s, create ui: 4.6s, gradio launch: 2.5s, app_started_callback: 0.3s).
*** Error running process: E:\Stable-Diffusion-webui-update\extensions\sd-dynamic-prompts\scripts\dynamic_prompting.py
    Traceback (most recent call last):
      File "E:\Stable-Diffusion-webui-update\modules\scripts.py", line 519, in process
        script.process(p, *script_args)
      File "E:\Stable-Diffusion-webui-update\extensions\sd-dynamic-prompts\sd_dynamic_prompts\dynamic_prompting.py", line 482, in process
        all_prompts, all_negative_prompts = generate_prompts(
      File "E:\Stable-Diffusion-webui-update\extensions\sd-dynamic-prompts\sd_dynamic_prompts\helpers.py", line 93, in generate_prompts
        all_prompts = prompt_generator.generate(prompt, num_prompts, seeds=seeds) or [""]
      File "E:\Stable-Diffusion-webui-update\venv\lib\site-packages\dynamicprompts\generators\randomprompt.py", line 71, in generate
        prompts.append(next(iter(gen)))
    StopIteration

---
2023-08-09 12:33:24,035 - ControlNet - INFO - Loading model: control_v11f1e_sd15_tile [a371b31b]
2023-08-09 12:33:24,459 - ControlNet - INFO - Loaded state_dict from [E:\Stable-Diffusion-webui-update\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.pth]
2023-08-09 12:33:24,460 - ControlNet - INFO - Loading config: E:\Stable-Diffusion-webui-update\extensions\sd-webui-controlnet\models\control_v11f1e_sd15_tile.yaml
2023-08-09 12:33:26,837 - ControlNet - INFO - ControlNet model control_v11f1e_sd15_tile [a371b31b] loaded.
2023-08-09 12:33:26,921 - ControlNet - INFO - Loading preprocessor: tile_resample
2023-08-09 12:33:26,921 - ControlNet - INFO - preprocessor resolution = 512
2023-08-09 12:33:26,943 - ControlNet - INFO - Loading model: control_v11p_sd15_softedge [a8575a2a]
2023-08-09 12:33:27,403 - ControlNet - INFO - Loaded state_dict from [E:\Stable-Diffusion-webui-update\extensions\sd-webui-controlnet\models\control_v11p_sd15_softedge.pth]
2023-08-09 12:33:27,403 - ControlNet - INFO - Loading config: E:\Stable-Diffusion-webui-update\extensions\sd-webui-controlnet\models\control_v11p_sd15_softedge.yaml
2023-08-09 12:33:29,600 - ControlNet - INFO - ControlNet model control_v11p_sd15_softedge [a8575a2a] loaded.
2023-08-09 12:33:29,691 - ControlNet - INFO - Loading preprocessor: pidinet
2023-08-09 12:33:29,691 - ControlNet - INFO - preprocessor resolution = 512
2023-08-09 12:33:29,716 - ControlNet - INFO - Loaded state_dict from [E:\Stable-Diffusion-webui-update\extensions\sd-webui-controlnet\annotator\downloads\pidinet\table5_pidinet.pth]
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:09<00:00,  2.18it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:08<00:00,  2.34it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [00:08<00:00,  2.36it/s]

Additional information

No response

[catboxanon]

I can replicate it but I'm not fully convinced it's a webui issue yet.

I repeat, everything worked correctly before and I have a backup copy of AUTOMATIC1111 that has not been updated for several months and everything works correctly there!!! And it seems to me that the problem is in AUTOMATIC 1111, since updating ControlNet to the latest version in my backup did not lead to an error, but updating AUTOMATIC1111 led to its occurrence.

Can you provide the commit hash for your backed up webui version? As well as the old ControlNet version? If so I can do a bisect of this to see if this is true or not.

[catboxanon]

Actually, I see what the problem is now... it is, however, not the webui's fault per se. I'll open a PR upstream soon for the extension.

[catboxanon]

Mikubill/sd-webui-controlnet#1907

[CrazyMaxTM]

Я могу воспроизвести это, но я еще не полностью уверен, что это проблема с webui.

Повторяю, раньше все работало корректно, и у меня есть резервная копия AUTOMATIC1111, которая не обновлялась в течение нескольких месяцев, и там все работает корректно!!! И мне кажется, что проблема в AUTOMATIC 1111, поскольку обновление ControlNet до последней версии в моей резервной копии не привело к ошибке, но обновление AUTOMATIC1111 привело к ее возникновению.

Можете ли вы предоставить хэш фиксации для вашей резервной версии webui? А также для старой версии ControlNet? Если да, я могу разделить это пополам, чтобы увидеть, верно это или нет.

My backup has a hash 22bcc7b

I was also able to update it to cf28aed
everything still worked correctly there, but after a few commits after (I don’t remember exactly how many) it broke ...

[catboxanon]

Thanks -- as I mentioned though I actually did figure this out now. I made a PR (#12470) that is available on the dev branch and will be available for new versions soon, and also a PR for ControlNet (Mikubill/sd-webui-controlnet#1907) that would make it compatible for versions with and without the aforementioned PR.