Discard cache based on file age

[allefeld]

I want to use bkt to cache preprocessor results, like the role lessfile / lesspipe plays for less. Important here is that the cache is renewed when the file has changed. --modtime helps in principle, but has the disadvantage that cache entries stick around even though they will never be used again (if I understand correctly how the option works). My workaround is to set --ttl to the file age.

A simple proof of concept, caching md5sum:

#!/bin/bash
file=$(realpath "$1")
mtime=$(stat -c %Y "$file")
now=$(date +%s)
age=$(( now - mtime ))
bkt --ttl "${age}s" -- md5sum "$file"

I'd be interested to know whether this indeed works as intended, or can be improved.

[dimo414]

Hey @allefeld thanks for the post! I think --modtime doesn't work quite how you're picturing it, but I might also be misunderstanding your requirements.

The example above, setting a TTL based on a file's "age", will cause the cache to be invalidated at some point in the future (e.g. if the file was last edited yesterday, you're effectively passing --ttl 24h) which sort of works as a heuristic for "how likely the file will remain unmodified going forward", but otherwise is rather arbitrary and I'm not sure what it affords you. Perhaps you can help me understand how you want to use this?

By contrast --modtime affects the cache key, not its validity duration. As soon as a file is modified every cached command associated with the prior --modtime becomes invalidated, and as long as the file is unchanged (regardless of its age) the cache remains valid.

There is definitely some room for me to improve cache cleanup, including what's discussed in #27. If this is a motivating concern for you, or if there's a different class of cache leak you're observing, I should be able to take a swing at it before the end of the year.

[allefeld]

Here's my use case:

I have written a preprocessor (as a shell script) which turns binary files of some types into readable text. This preprocessor can be used e.g. with less, so I can view e.g. the contents of a Word file in the terminal, and it can be used with ripgrep, so I can search through a collection of e.g. Word files. This preprocessing can take a bit of time. Not too much for individual file viewing with less, but for a search through dozens to hundreds of files with ripgrep, it adds up to something substantial.

So my idea was to wrap the call to my preprocessor in bkt, to cache the results of the preprocessing. This cache does not have a fixed time limit – as long as the original file is unchanged, the preprocessor output will also remain unchanged. Since bkt caches arbitrary command output, I need to communicate to bkt my cache retention / discard rule.

Now it sounds like --modtime is meant for this. However, I don't understand the strategy of including the time in the cache key. Yes, one gets invalidation this way – but also the previous cache file, which will never be used again, sticks around until it is cleaned up separately. To me it makes more sense not to include the time in the key, so the cache file name can be reused, but an old cache file is not used if it is too old and instead overwritten by the new output of the preprocessor. That is what I tried to achieve via --ttl.

In the meantime I implemented my own simple cache mechanism within my shell script, and it seems to be working well. The cache file name is based on a hash of the original file path. It is used if it is newer than the original file and overwritten otherwise. So maybe there's no need to try and make bkt do what I want?