OOM; What is required for this to run. *Linux, 2080Ti 11Gb seems not to cut it

[somenewaccountthen]

How much VRAM is needed and could you specify it on the main page please.

(mangaNinjia) /my_dev/big_ai_etc/imaging/mangaNinjia/scripts$ ./infer.sh

• pretrained_model_name_or_path=./checkpoints/StableDiffusion
• image_encoder_path=./checkpoints/models/clip-vit-large-patch14
• controlnet_model_name_or_path=./checkpoints/models/control_v11p_sd15_lineart
• annotator_ckpts_path=./checkpoints/models/Annotators
• manga_reference_unet_path=./checkpoints/MangaNinjia/reference_unet.pth
• manga_main_model_path=./checkpoints/MangaNinjia/denoising_unet.pth
• manga_controlnet_model_path=./checkpoints/MangaNinjia/controlnet.pth
• point_net_path=./checkpoints/MangaNinjia/point_net.pth
• export CUDA_VISIBLE_DEVICES=0
• CUDA_VISIBLE_DEVICES=0
• input_reference_paths='./test_cases/hz0.png ./test_cases/hz1.png'
• input_lineart_paths='./test_cases/hz1.png ./test_cases/hz0.png'
• point_ref_paths='./test_cases/hz01_0.npy ./test_cases/hz01_1.npy'
• point_lineart_paths='./test_cases/hz01_1.npy ./test_cases/hz01_0.npy'
• cd ..
• python infer.py --seed 0 --denoise_steps 50 --pretrained_model_name_or_path ./checkpoints/StableDiffusion --image_encoder_path ./checkpoints/models/clip-vit-large-patch14 --controlnet_model_name_or_path ./checkpoints/models/control_v11p_sd15_lineart --annotator_ckpts_path ./checkpoints/models/Annotators --manga_reference_unet_path ./checkpoints/MangaNinjia/reference_unet.pth --manga_main_model_path ./checkpoints/MangaNinjia/denoising_unet.pth --manga_controlnet_model_path ./checkpoints/MangaNinjia/controlnet.pth --point_net_path ./checkpoints/MangaNinjia/point_net.pth --output_dir output --guidance_scale_ref 9 --guidance_scale_point 15 --input_reference_paths ./test_cases/hz0.png ./test_cases/hz1.png --input_lineart_paths ./test_cases/hz1.png ./test_cases/hz0.png --point_ref_paths ./test_cases/hz01_0.npy ./test_cases/hz01_1.npy --point_lineart_paths ./test_cases/hz01_1.npy ./test_cases/hz01_0.npy
  /home/admin_beppie/anaconda3/envs/mangaNinjia/lib/python3.10/site-packages/diffusers/models/dual_transformer_2d.py:20: FutureWarning: DualTransformer2DModel is deprecated and will be removed in version 0.29. Importing DualTransformer2DModel from diffusers.models.dual_transformer_2d is deprecated and this will be removed in a future version. Please use from diffusers.models.transformers.dual_transformer_2d import DualTransformer2DModel, instead.
  deprecate("DualTransformer2DModel", "0.29", deprecation_message)
  INFO:root:output dir = output
  arguments: Namespace(output_dir='output', denoise_steps=50, seed=0, pretrained_model_name_or_path='./checkpoints/StableDiffusion', image_encoder_path='./checkpoints/models/clip-vit-large-patch14', controlnet_model_name_or_path='./checkpoints/models/control_v11p_sd15_lineart', annotator_ckpts_path='./checkpoints/models/Annotators', manga_reference_unet_path='./checkpoints/MangaNinjia/reference_unet.pth', manga_main_model_path='./checkpoints/MangaNinjia/denoising_unet.pth', manga_controlnet_model_path='./checkpoints/MangaNinjia/controlnet.pth', point_net_path='./checkpoints/MangaNinjia/point_net.pth', input_reference_paths=['./test_cases/hz0.png', './test_cases/hz1.png'], input_lineart_paths=['./test_cases/hz1.png', './test_cases/hz0.png'], point_ref_paths=['./test_cases/hz01_0.npy', './test_cases/hz01_1.npy'], point_lineart_paths=['./test_cases/hz01_1.npy', './test_cases/hz01_0.npy'], is_lineart=False, guidance_scale_ref=9.0, guidance_scale_point=15.0)
  INFO:root:device = cuda
  /home/admin_beppie/anaconda3/envs/mangaNinjia/lib/python3.10/site-packages/diffusers/models/lora.py:306: FutureWarning: LoRACompatibleConv is deprecated and will be removed in version 1.0.0. Use of LoRACompatibleConv is deprecated. Please switch to PEFT backend by installing PEFT: pip install peft.
  deprecate("LoRACompatibleConv", "1.0.0", deprecation_message)
  Some weights of the model checkpoint at ./checkpoints/StableDiffusion were not used when initializing RefUNet2DConditionModel: ['conv_out.weight', 'conv_norm_out.weight', 'conv_norm_out.bias', 'conv_out.bias']

• This IS expected if you are initializing RefUNet2DConditionModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing RefUNet2DConditionModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
  /home/admin_beppie/anaconda3/envs/mangaNinjia/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default. For more details check this issue: [BUG] GPT-2 tokenizer is NOT invertible huggingface/transformers#31884
  warnings.warn(
  Traceback (most recent call last):
  File "/my_dev/big_ai_etc/imaging/mangaNinjia/infer.py", line 222, in
  pipe = pipe.to(torch.device(device))
  File "/home/admin_beppie/anaconda3/envs/mangaNinjia/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 418, in to
  module.to(device, dtype)
  File "/home/admin_beppie/anaconda3/envs/mangaNinjia/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1173, in to
  return self._apply(convert)
  File "/home/admin_beppie/anaconda3/envs/mangaNinjia/lib/python3.10/site-packages/torch/nn/modules/module.py", line 779, in _apply
  module._apply(fn)
  File "/home/admin_beppie/anaconda3/envs/mangaNinjia/lib/python3.10/site-packages/torch/nn/modules/module.py", line 779, in _apply
  module._apply(fn)
  File "/home/admin_beppie/anaconda3/envs/mangaNinjia/lib/python3.10/site-packages/torch/nn/modules/module.py", line 779, in _apply
  module._apply(fn)
  [Previous line repeated 2 more times]
  File "/home/admin_beppie/anaconda3/envs/mangaNinjia/lib/python3.10/site-packages/torch/nn/modules/module.py", line 804, in _apply
  param_applied = fn(param)
  File "/home/admin_beppie/anaconda3/envs/mangaNinjia/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1159, in convert
  return t.to(
  torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB. GPU