Tag memory consumption and SetSize

[tomwarner13]

We're migrating a huge application which was previously using a hand-rolled L2 cache system that got put together in haste some time in the mid-2010s, to FusionCache. Overall it's been a good experience (I especially appreciate the comprehensize docs!), but there is something we're running into that I want to understand.

To preserve some of the legacy compatibility, we are currently using a LOT of tags. Like, potentially 10-12 tags per every "bucket" of a few dozen entries, to grotesquely oversimplify. We noticed that the application using FusionCache on production servers will consistently chew through all its available memory and start thrashing to disk if not restarted regularly. We attempted to set up a SizeLimit on the MemoryCache object that FusionCache is using, and set a default size of 1 on every FusionCache entry, but this had no effect.

I've been investigating more and I think the issue is our tagging implementation. We do ultimately plan to refactor out most of the tagging logic and consolidate down to far fewer tags, but that is a longer-term effort.

Meanwhile, what I have been seeing is that every time we set a tag, FusionCache adds this as a new entry to the MemoryCache with a Size of 0 (even when the default size for other entries is >0), a long absolute expiration (like 2 weeks in the future), and a priority of NeverRemove. If I understand what I'm seeing correctly, this means that memcache is continuing to fill its capacity up with tag items as the application runs, and even when we attempt to compact items out due to memory pressure, none of these tags will ever be evicted and eventually we have to restart the server and start over.

So I have a few questions around this:

• Do I understand this behavior correctly that every tag which is set on any entry leads to a MemoryCache entry which can never disappear short of a deliberate key-targeted clear or disposal of the whole cache?
• Is there any other best practice or way around this other than "use less tags"? (We're working toward that I promise 🙃)
• Should this be intended behavior? There's always a possibility that the entry disappears from memory cache anyway due to a restart or something so I imagine there needs to be default handling (calling L2 if available, etc) for tags which are missing from memory cache - could we configure something that allows them to not eat memory forever?

[tomwarner13]

.....and now I see there's a TagsDefaultEntryOptions that I can probably leverage to fix at least some of this, looking into whether it fixes our issues.

[tomwarner13]

It does appear that setting TagsDefaultEntryOptions lets us manage tag duration/expiration a lot more effectively, I think there's nothing more to follow up on here.

[jodydonetti]

Hi @tomwarner13 , sorry for the delay.

We're migrating a huge application which was previously using a hand-rolled L2 cache system that got put together in haste some time in the mid-2010s, to FusionCache. Overall it's been a good experience (I especially appreciate the comprehensize docs!)

Great, happy to hear that!

but there is something we're running into that I want to understand.

[...]
Meanwhile, what I have been seeing is that every time we set a tag, FusionCache adds this as a new entry to the MemoryCache

Yes and no, but nitpicking here: the "special entries" for each tag are added not when saving an entry that has tags attached, but when reading that entry later.

So, simple example: you save an entry with 10 tags, but then never access it again? No special entries saved in the cache.

Also, this is true for L1 (memory level), but if you also use and L2 (distributed level) then there it's different: the special entries will only be created in the L2 if you do a RemoveByTag() call, otherwise no cache pollution with extra special entries.

with a Size of 0 (even when the default size for other entries is >0), a long absolute expiration (like 2 weeks in the future), and a priority of NeverRemove. If I understand what I'm seeing correctly, this means that memcache is continuing to fill its capacity up with tag items as the application runs, and even when we attempt to compact items out due to memory pressure, none of these tags will ever be evicted and eventually we have to restart the server and start over.

Yes, it's probably an oversimplification but yes, that's the case.

And as you already discovered in the next comment, the entry options for those special entries are specified either:

• directly: when calling RemoveByTag(tag, options)
• tags-specific defaults: in the TagsDefaultEntryOptions which you can change during setup (see here)

But watch out: the special entries for the tags are strictly needed for Tagging to work. If you remove them or let them disappear before the right moment, you may have zombie entries (theoretically removed by a RemoveByTag() call) suddenly reappear later.

For a better understanding of how Tagging work in FusionCache and why I took certain design decision, I would suggest a read here or here.

So I have a few questions around this:

• Do I understand this behavior correctly that every tag which is set on any entry leads to a MemoryCache entry which can never disappear short of a deliberate key-targeted clear or disposal of the whole cache?

Kind of, yes (plus the additional bits of info mentioned above).

• Is there any other best practice or way around this other than "use less tags"? (We're working toward that I promise 🙃)

Ahaha yes I believe you, but 10-12 tags per entry is generally sustainable.

In general the advice with Tagging is "do not overtag", that's it.
And no, sadly there is not a precise amount I can give, as it depends on a lot of factors.
FusionCache already does a lot of internal optimizations to make it work as best as possible, but still that remains as a good baseline advice.

(also: it's very similar to the main advice you can find with observability/OTEL and metrics, so it's kind of an industry standard)

• Should this be intended behavior? There's always a possibility that the entry disappears from memory cache anyway due to a restart or something so I imagine there needs to be default handling (calling L2 if available, etc) for tags which are missing from memory cache - could we configure something that allows them to not eat memory forever?

No, in theory it should work all the times no matter what, let me explain.
We have 2 main scenarios/setup: L1 only and L1+L2.

Here's how it works:

• L1 only: if the app restarts, the cache becomes empty (a.k.a. cold start). In this case there's no need for the tagging-related special entries anymore, since... everything disappeared anyway
• L1+L2: in this case (both single node and multi-node) a restart would wipe the L1, but not the L2. And anything not present in the L1 (cache miss) will be automatically take from L2, so everything should work perfectly fine

Makes sense?

Hope this helps, let me know.

[jodydonetti]

Also, I'd like to add a concrete example, just to be more clear.

Let's say you specify, via TagsDefaultEntryOptions, that tagging-related special entries should have a Duration of, say, 10sec.

Then you save an entry with a cache key "foo" and a tag "my-tag", with a Duration of 5min.

Then you do a RemoveByTag("my-tag"): here, internally, the special entry with key "__fc:t:my-tag" is created, and it contains the current timestamp (meaning the timestamp of the RemoveByTag() operation).

Now, if you try to get the entry "foo", FusionCache will internally find it in the cache (L1, L2, it doesn't matter) but it will also notice that a RemoveByTag() operation happened, for one of its tags, after the entry was created. So it will ignore the entry and have a logical cache miss.

But if you instead wait for 10sec and then you get "foo", the entry will be used, since the information about the RemoveByTag() operation went lost.

Hope this helps.