Added optional argument to store model hashes

[radiopabm2]

Description

What this PR do?

This PR add support for local hashes storing. It is very usable when using remote models storage, such as S3 mounted through s3fs or using very large models storages. Significantly reduces startup speed but needs to "compile" hashes on first launch.
It is not happens until you provide "--local-hash-path" argument.
In my configuration with local s3 storage on 10Gb\sec network reduces "list SD models" time on startup from 617.0s to 0.4s.

Why it is needed?

When using remote model storages stable-diffusion-webui models files will be fully downloaded on the local system to generate hash. When using large local model storages it will read every model file to generate hash.
Due to this specific behavior, startup time can increase significantly when using remote storage, or just a HUGE local storage, where large amount of big files must be readen to generate hash. This PR reduces startup speed with optional storing models hashes. Also potentially it can preserve model spoof attacks too: hashes generated earlier safer than hashes that updates every launch.

This PR changes:

modules/cmd_args.py:

• appending new argument --local-hash-path

modules/sd_models.py:

• added variable models_hash_path to store path from commandline
• appending new function model_hash_generate that generates and reads hash files with given path
• changing behaviour of model_hash function to check property models_hash_path is defined and choose what to do: generate now or use stored hashes.

Final behaviour:

When launching with --local-hash-path: on every call of model_hash it will check existing of hash file and given directory. If it is exists - just reads hash. If not - creating dir (if not exists) and file, generating hash and writing it.

Without --local-hash-path: behaviour not changing at all.

Checklist:

• I have read contributing wiki page
• I have performed a self-review of my own code
• My code follows the style guidelines
• My code passes tests

[w-e-w]

nice catch, good PR
but we actually have a system for cacheing things such as hashes
and we would also compare the modification time of the file to decide whether or not the hash needs to be recalculated
I will make another PR and credit you as a co-author

[radiopabm2]

nice catch, good PR but we actually have a system for cacheing things such as hashes and we would also compare the modification time of the file to decide whether or not the hash needs to be recalculated I will make another PR and credit you as a co-author

Thank you for fast answer and pr review!
Is there more information about existing caching system or it is WIP? I was try but not found anything. Also checking modification time feature not too hard to do - i can commit this pr or just start\append other if you want.

[w-e-w]

Is there more information about existing caching system or it is WIP?

not WIP, it's very old stuff
we calculate the hash using our hashes.py module, whitch uses cache.py

we forgot about the old hash as it was deprecated for a long time

we used to store all hash in cache.json file, but but it became problematic and we switched to using diskcache (basically a sqlite wrapper)

• use diskcache library for caching #15287

[radiopabm2]

we been cacheing things for a long time see cache.py
we calculate the hash using our hashes.py module, whitch uses cache.py
we forgot about the old hash as it was deprecated for a long time
he used to store all hash in json, then we switch to using diskcache (basically a sqlite wrapper)

Thanx for explanation! It seeems you can close this PR now. Will wait new pr will be submitted :)

[w-e-w]

btw
you can pull #16830 to your own brunch / server / local instance for your own use if you want it use it now and not have to wait