Current code for dependencies checking is causing infinite package reinstalls and other inconsistencies

[illtellyoulater]

What happened?

So many extensions (even popular ones like TemporalKit) are messing up python environments because the wrong way they are checking for dependencies at runtime.
Consider the example code in the wiki (https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Developing-extensions#installpy) for installing an extension, which gets executed at every launch:

import launch

if not launch.is_installed("aitextgen"):
    launch.run_pip("install aitextgen==0.6.0", "requirements for MagicPrompt")

This breaks for all libraries having a "-" in their name for example ffmpeg-python, scikit-image, etc... which are then installed even when they already are.

As of now the majority of extensions are just iterating over their requirements.txt file with this code... Imagine the inconsistencies this is causing...

Please just add a note to the wiki section about this !

Steps to reproduce the problem

> launch_utils.run_pip("install ffmpeg-python", "required package")
[ Installing required package: "ffmpeg-python" ... ]
[ Installed ]

> launch_utils.is_installed("ffmpeg-python")
False
> launch_utils.is_installed("ffmpeg")
True

What should have happened?

Extension authors reading the wiki should have been reminded to not iterate over requirements.txt using the above code.

Sysinfo

n/a

What browsers do you use to access the UI ?

No response

Console logs

n/a

Additional information

No response

[missionfloyd]

It's not necessarily that there's hyphens, it's that module names don't always match package names.

> launch_utils.is_installed("pillow")
False
> launch_utils.is_installed("PIL")
True

It's possible to check package names like this.

import importlib.metadata
import launch

def is_installed(package):
    try:
        dist = importlib.metadata.distribution(package)
    except importlib.metadata.PackageNotFoundError:
        return False

    return dist is not None

if not is_installed("ffmpeg-python"):
    launch.run_pip("install ffmpeg-python", "requirements")

[illtellyoulater]

Yes, correct! Not just about the hyphens! (what was I thinking? lol)

So I replaced the current is_installed() function from launch_utils.py with your code above, disabled all extensions (even built-in ones), relanched webui.sh and now finally the presence of extensions dependancies are correctly verified and they are not reinstalled.

However as you can see now there are 2 other reinstall happening at an earlier stage (open_clip and requirements)

################################################################
Launching launch.py...
################################################################
Using TCMalloc: libtcmalloc_minimal.so.4
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
Version: v1.6.0
Commit hash: 5ef669de080814067961f28357256e8fe27544f4
Installing open_clip  <-----------
Installing requirements  <-----------
You are up to date with the most recent release.
Launching Web UI with arguments: --xformers --update-check --theme dark --port 7860
*** "Disable all extensions" option was set, will not load any extensions ***
Loading weights [6ce0161689] from /opt/stablediffusion-webui/models/Stable-diffusion/v1-5-pruned-emaonly.safetensors
Running on local URL:  http://127.0.0.1:7860

This happens when "lauch_utils.prepare_enviroment()" from launch_utils.py is called from launch.py. which is where is_installed() is also called for the first time.
However in this case your new implementation fails because it has to deal with a zip file and a requirements file, don't have any "distributiion" property:

def prepare_environment():
     openclip_package = os.environ.get('OPENCLIP_PACKAGE', "https://github.com/mlfoundations/open_clip/archive/bb6e834e9c70d9c27d0dc3ecedeebeaeb1ffad6b.zip")

    if not is_installed("open_clip"):   
        run_pip(f"install {openclip_package}", "open_clip")      #  <------------
   
   ...
   
    if not requirements_met(requirements_file):  
        run_pip(f"install -r \"{requirements_file}\"", "requirements")       #   <------------
        startup_timer.record("install requirements")

Anyway, specific cases apart, I can confirm that with the new version of is_installed(), all previously failing dependencies checks and reinstalls coming from extensions are finally gone!

[illtellyoulater]

Could we have something like this to account for different cases?

def is_installed(package):
    try:
    	dist = importlib.metadata.distribution(package)
    except importlib.metadata.PackageNotFoundError:
        try: 
            spec = importlib.util.find_spec(package)
        except ModuleNotFoundError:
    	    return False

        return spec is not None

    return dist is not None

I'm currently testing and it seems to work fine... but I was wondering if this approach comes with some secuirity implications? Giving extensions too much power?

[missionfloyd]

Openclip's package is open-clip-torch, and requirements_met() doesn't use is_installed().

[illtellyoulater]

Indeed requirements_met() does not use is_installed(), but it behaves similarly to it, making the same mistake, i.e. calling importlib.metadata.version directly on package rather than first determining its correct name.

def requirements_met(requirements_file):
      
    try:
        version_installed = importlib.metadata.version(package)
    except Exception:
        return False

EDIT: it's been 2 months and I can confirm that so far the change shown in previous post has been working flawlessly.