`trial` test runs leak memory

[squahtx]

#15334 introduced a bug where HomeServer instances no longer get garbage collected during trial test runs. As a result, memory usage grows to multiple gigabytes during test runs.

See https://github.com/matrix-org/synapse/pull/15334/files#diff-f50ec854a65082453f4086048f20c291ca6020fab973486a9da2285025394cc8R180.
When we garbage collect generation 0, self.hs still holds a reference to the HomeServer, which gets promoted to generation 1 and then never collected.

Apart from that, there are also a couple of other leaks at play.

---

• Fix HomeServers leaking due to not being GCed.
  Fix HomeServers leaking during trial test runs #15630
• Fix FederationRateLimiter (and referenced HomeServers) leaking due to _rate_limiter_instances global.
  Fix HomeServers leaking during trial test runs #15630
• Fix config cache added in Speedup tests by caching HomeServerConfig instances #15284 growing too large (~360 entries). HomeServerConfigs are pretty heavy and hold on to a lot of stuff, including Jinja templates. A suitable max size for the cache might be 8-16 entries.
  Limit the size of the HomeServerConfig cache in trial test runs #15646
• Fix HomeServers leaking via synapse.util.caches.caches_by_name.
• Fix HomeServers leaking via synapse.util.caches.CACHE_METRIC_REGISTRY.
• Fix HomeServers leaking via synapse.config.cache._CACHES.

The last 3 are fairly annoying to fix, so I've left them alone.

[squahtx]

When I force a full GC every 50 or so gen0 GCs, a trial run no longer OOMs on my laptop. But memory usage still keeps growing and the GCs get slower and slower. So there's a separate memory leak going on.

According to tracemalloc, the most leaked objects are

    /home/squah/repos/synapse/synapse/util/caches/lrucache.py:483: size=26.2 MiB, count=217769, average=126 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/jinja2/environment.py:704: size=23.5 MiB, count=183329, average=134 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/jinja2/environment.py:1299: size=16.5 MiB, count=1515, average=11.2 KiB
    /home/squah/repos/synapse/synapse/storage/database.py:469: size=11.2 MiB, count=140743, average=83 B
    /usr/lib/python3.10/functools.py:58: size=10.3 MiB, count=168891, average=64 B
    /usr/lib/python3.10/functools.py:52: size=8257 KiB, count=93189, average=91 B
    /usr/lib/python3.10/linecache.py:137: size=6795 KiB, count=67319, average=103 B
    /home/squah/repos/synapse/synapse/util/caches/lrucache.py:759: size=4862 KiB, count=8643, average=576 B
    /home/squah/repos/synapse/synapse/storage/background_updates.py:547: size=4747 KiB, count=42392, average=115 B
    /home/squah/repos/synapse/synapse/config/_base.py:168: size=4571 KiB, count=55214, average=85 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/jinja2/environment.py:349: size=4557 KiB, count=4106, average=1137 B
    /home/squah/repos/synapse/synapse/util/caches/deferred_cache.py:108: size=4185 KiB, count=105808, average=40 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/twisted/internet/base.py:114: size=3669 KiB, count=12692, average=296 B
    /home/squah/repos/synapse/synapse/util/caches/lrucache.py:502: size=2971 KiB, count=25931, average=117 B
    /home/squah/repos/synapse/synapse/storage/database.py:192: size=2807 KiB, count=35984, average=80 B
    /usr/lib/python3.10/unittest/mock.py:415: size=2564 KiB, count=5466, average=480 B
    /home/squah/repos/synapse/synapse/storage/background_updates.py:509: size=2560 KiB, count=19521, average=134 B
    /usr/lib/python3.10/functools.py:61: size=2388 KiB, count=14553, average=168 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/jinja2/environment.py:350: size=2359 KiB, count=4108, average=588 B
    /home/squah/repos/synapse/.venv/lib/python3.10/site-packages/twisted/internet/defer.py:461: size=2306 KiB, count=29910, average=79 B

but this isn't too helpful in tracking down the leak.

[squahtx]

One of the leaks is due to

synapse/synapse/util/ratelimitutils.py

Line 89 in a302d3e

_rate_limiter_instances: Set["FederationRateLimiter"] = set()

Another "leak" is due to the config cache added in #15284 growing large (~80 entries).

[squahtx]

Another "leak" is due to the config cache added in #15284 growing large.

I'm going to leave this one alone. I'd replace things with functools.lru_cache, except config dicts are unhashable and there's no way to give lru_cache a custom cache key.

[DMRobertson]

except config dicts are unhashable

FWIW I think immutabledict (previously frozendict) is hashable?

[squahtx]

except config dicts are unhashable

FWIW I think immutabledict (previously frozendict) is hashable?

That's true, we could frozendictify the config dicts and use them as cache keys. I'd balk at the copying involved, but the current code does a json.dumps to derive the cache key every time it's called.