SQLite for cache

[w-e-w]

Description

as opposed to using json file use a SQLite database of cache

reason of this because we have seen some users with extremely large cache files
like catboxanon cache.json is > 236MB
it's becomes increasingly impractical to use it as a method of storage
and we have seen some cases of the the cache.json beeing being corruptedfor unknown reason reasons #11773

the data will not be stored under cache_database.db or the path specified by env SD_WEBUI_CACHE_DATABASE

to make as little changes as possible I essentially implemented a translation layer that essentially represents the database as a python object same as if we load and save a json file
assuming that I have done everything correctly it should be basically transparent
the implementation is meant to be simple, but this also means not truly optimized for an SQL database
I wrote json strings into the value column for the table for each individual entries
this would mean they will be a slight overhead when reading and writing to the database to do the json conversion

currently this is implemented as a toggle which can be changed under settings > system > Use sqlite for cache
I store the value under the key experimental_sqlite_cache if this is proven stable with no compatibility issues we can remove the enable this by default in the future

Other changes

I changed the file modification time comparison for that triggers a cache refresh from greater than to not equal
this makes sense because you might replace a new file with another file that has a older modification date
if this is the case then the cache refresh will not be triggered, this is undesirable

TODO

if in the future this is going to be set at as the default
it should be a simple matter to write a migration script to migrate cache.json to the db so that users don't have to spend time recalculating

Database table structure

Table: hashes

path (string)	mtime (float)	value (json string)
checkpoint/Drawing\Stable Diffusion\v1-5-pruned-emaonly.ckpt	1666314275.47775	"cc6cb2710......" (the full hash)
......	......	......

Table: safetensto-metadata

path (string)	mtime (float)	value (json string)
checkpoint/Drawing\Stable Diffusion\sd_xl_base_1.0.safetensors	1690539434.22002	{"modelspec.sai_model_spec": "1.0.0", ...... } (json meta date)
......	......	......

this will be read into memory as one big object, structure mimicking cache.json

cache_data = {
    "hashes": {
        "checkpoint/Drawing\Stable Diffusion\v1-5-pruned-emaonly.ckpt": {
            "mtime": 1666314275.47775,
            "value": "cc6cb2710......"  # (the full hash)
        },
        { ...... }
    },
    "safetensto-metadata": {
        "checkpoint/Drawing\Stable Diffusion\sd_xl_base_1.0.safetensors": {
            "mtime": 1690539434.22002,
            "modelspec.sai_model_spec": "1.0.0",
            # ......
        },
        { ...... }
    }
}

Checklist:

• I have read contributing wiki page
• I have performed a self-review of my own code
• My code follows the style guidelines
• My code passes tests

[wfjsw]

I'm reading it on mobile phone so I'm probably missing things

doing r/w on full file scale probably won't help with the performance. i might be able to come up with something that is better but only on weekends

from my experience, anything other than full wal is unreliable. it has to be enabled for sqlite to be robust against corruptions

[w-e-w]

@wfjsw
I think you probably misread
I'm only doing full read but not full write

I read from the entire database once at the launch of web UI
and this is what we are doing with the json, this is faster than querying the entries individually as needed
however this is less memory efficient, so if we want we can change this easily

subsequent write if any are only per entry, it does not be rewrite entire database
so it should perform better as data size get larger

[freecoderwaifu]

Can confirm, almost 20k Loras (send help).

[w-e-w]

wow 500MB 🤯
help test if you can

[freecoderwaifu]

Finally got around to testing it

Nice size reduction, startup took a while to create the new database though, unsure how long since I AFK'd while it was doing it, but .json cache rebuild times are also long.

Everything else seems working exactly as with the older cache, fresh launch after a (re)boot takes about the same time but "seemed" a tiny bit faster, subsequent relaunches are also fast as before.

[w-e-w]

duction, startup took a while to create the new database though, unsure how long since I AFK'd while it was doing it, but .json cache rebuild times are also long.

when and if this is merge I intend to write a migration sequence that would use the json file build the database

because currently I consider this to be in "experimental" phase I didn't want to add extra logic fot migration yet

the migration logic should be relatively simple
if sqlite_cache and cache.json exist and sqlite_cache.db not exist
create db and dump the contents of cache.json into each individual tables in database

Everything else seems working exactly as with the older cache, fresh launch after a (re)boot takes about the same time but "seemed" a tiny bit faster, subsequent relaunches are also fast as before.

the main benefit of this should be when writing new cache it doesn't have to rewrite the entire file
then they should be far lesser chance of the database being corrupted
which if this happens means rebuilding of the entire cache, like what we have seen happen to some users

[akx]

TBH I think we should just use https://pypi.org/project/diskcache/ instead. It's well battle-tested, and SQLite (+disk for large objects) backed. Also, for the app it looks just like a dict, which is very useful.

[w-e-w]

cool, didn't know about diskcache
that's why I just put together my own implementation

[AUTOMATIC1111]

merged #15287