Fix looping calls not getting GCed. #19416
Merged
erikjohnston merged 4 commits intodevelopfrom Jan 30, 2026
Merged
Conversation
The `Clock` tracks looping calls to allow cancelling of all looping calls. However, this stopped them from getting garbage collected.
That way when we log looping calls it points to the actual function, rather than the wrapper.
devonh
approved these changes
Jan 29, 2026
Member
devonh
left a comment
devonh
left a comment
There was a problem hiding this comment.
Ah - dang. Thanks for cleaning up this oversight of mine :)
Changes look good.
| if now: | ||
| looping_call_context_string = "looping_call_now" | ||
|
|
||
| @wraps(f) |
| @@ -0,0 +1 @@ | |||
| Fix memory leak caused by not cleaning up stopped looping calls. Introduced in v1.140.0. | |||
Contributor
There was a problem hiding this comment.
How did you stumble upon this?
Investigation from seeing some memory graph?
Member
There was a problem hiding this comment.
I know it started with seeing the matrix.org client_readers take a long time to do GC, resulting in them appearing down frequently, https://github.com/matrix-org/internal-config/issues/1677
Then looking into why their memory usage was so high and kept growing.
Also, looking at this issue from the other day: #19392
I'm not sure how he jumped from there to looking at looping_calls.
Member
Author
There was a problem hiding this comment.
The basic steps were:
- Check the graphs to see if there was any obvious correlation, e.g. large cache size.
- Manhole into one of the workers and manually clear out all the caches, this did not help.
- Install
objgraphin the venv and runobjgraph.show_most_common_types(), this came back with a list ofcell,function(etc)... and thenLoopingCall. - At that point I had a look around trying to find where we keep references to looping calls, I just happened to stumble on
Clock._looping_calls. One can also use objgraph to generate graphs of references of objects to see what is holding on to the references, which also would have pointed toClock._looping_calls.
alexlebens
pushed a commit
to alexlebens/infrastructure
that referenced
this pull request
Feb 10, 2026
This PR contains the following updates: | Package | Update | Change | |---|---|---| | [element-hq/synapse](https://github.com/element-hq/synapse) | minor | `1.146.0` → `1.147.0` | --- ### Release Notes <details> <summary>element-hq/synapse (element-hq/synapse)</summary> ### [`v1.147.0`](https://github.com/element-hq/synapse/releases/tag/v1.147.0) [Compare Source](element-hq/synapse@v1.146.0...v1.147.0) ### Synapse 1.147.0 (2026-02-10) No significant changes since 1.147.0rc1. ### Synapse 1.147.0rc1 (2026-02-03) #### Bugfixes - Fix memory leak caused by not cleaning up stopped looping calls. Introduced in v1.140.0. ([#​19416](element-hq/synapse#19416)) - Fix a typo that incorrectly made `setuptools_rust` a runtime dependency. ([#​19417](element-hq/synapse#19417)) #### Internal Changes - Prune stale entries from `sliding_sync_connection_required_state` table. ([#​19306](element-hq/synapse#19306)) - Update "Event Send Time Quantiles" graph to only use dots for the event persistence rate (Grafana dashboard). ([#​19399](element-hq/synapse#19399)) - Update and align Grafana dashboard to use regex matching for `job` selectors (`job=~"$job"`) so the "all" value works correctly across all panels. ([#​19400](element-hq/synapse#19400)) - Don't retry joining partial state rooms all at once on startup. ([#​19402](element-hq/synapse#19402)) - Disallow requests to the health endpoint from containing trailing path characters. ([#​19405](element-hq/synapse#19405)) - Add notes that new experimental features should have associated tracking issues. ([#​19410](element-hq/synapse#19410)) - Bump `pyo3` from 0.26.0 to 0.27.2 and `pythonize` from 0.26.0 to 0.27.0. Contributed by [@​razvp](https://github.com/razvp) @​ ERCOM. ([#​19412](element-hq/synapse#19412)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4zLjYiLCJ1cGRhdGVkSW5WZXIiOiI0My4zLjYiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbImltYWdlIl19--> Reviewed-on: https://gitea.alexlebens.dev/alexlebens/infrastructure/pulls/3877 Co-authored-by: Renovate Bot <renovate-bot@alexlebens.net> Co-committed-by: Renovate Bot <renovate-bot@alexlebens.net>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
Clocktracks looping calls to allow cancelling of all looping calls. However, this stopped them from getting garbage collected.This was introduced in #18828
Fixes #19392