Warn about skipping reactor metrics when using unknown reactor type by MadLittleMods · Pull Request #19383 · element-hq/synapse

MadLittleMods · 2026-01-15T19:15:52Z

Warn about skipping reactor metrics when using unknown reactor type.

Spawning from not seeing any reactor metrics in the Grafana dashboard in some load tests, noticing python_twisted_reactor_tick_time_bucket is 0 in Prometheus, following it back to Synapse and seeing that we don't warn about skipping reactor metrics in all cases (when using an unknown reactor type).

A follow-up to this would be to actually figure out how to instrument the ProxiedReactor or why ProxiedReactor is being chosen in the first place and see if we can get it to use a more normal type 🤔 -> #19385

Reproduction instructions

Using the Complement scripts with workers: WORKERS=1 ./scripts-dev/complement.sh ./tests/csapi
docker logs complement_csapi_dirty_hs1 2>&1 | grep -i "reactor"
With these changes, notice Skipping configuring ReactorLastSeenMetric: unexpected reactor type: <__main__.ProxiedReactor object at 0x7fc0adaaea50> and Twisted reactor: ProxiedReactor
Cleanup:
- docker stop $(docker ps --all --filter "label=complement_context" --quiet)
- docker rm $(docker ps --all --filter "label=complement_context" --quiet)

I'm unable to reproduce with the normal Synapse images or complement-synapse without workers. They all use Twisted reactor: EPollReactor which is already supported.

Checking docker/Dockerfile-workers

Build the Docker image for Synapse: docker build -t matrixdotorg/synapse -f docker/Dockerfile . && docker build -t matrixdotorg/synapse-workers -f docker/Dockerfile-workers . (docs)

Start Synapse:

docker run -d --name synapse \
   --mount type=volume,src=synapse-data,dst=/data \
   -e SYNAPSE_SERVER_NAME=my.docker.synapse.server \
   -e SYNAPSE_REPORT_STATS=no \
   -e SYNAPSE_ENABLE_METRICS=1 \
   -p 8008:8008 \
   -p 9469:9469 \
   matrixdotorg/synapse-workers:latest

docker logs synapse 2>&1 | grep -i "reactor"
Says Twisted reactor: EPollReactor

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct (run the linters)

MadLittleMods · 2026-01-15T19:27:21Z

synapse/metrics/_reactor_metrics.py

            # E.g. this does not support the (Windows-only) ProactorEventLoop.
            logger.warning(
-                "Skipping configuring ReactorLastSeenMetric: unexpected asyncio loop selector: %r via %r",
+                "Skipping configuring reactor metrics: unexpected asyncio loop selector: %r via %r",


Made this generic as "reactor metrics" instead of ReactorLastSeenMetric because we're actually configuring python_twisted_reactor_last_seen AND python_twisted_reactor_tick_time here

MadLittleMods · 2026-01-15T19:28:01Z

synapse/metrics/_reactor_metrics.py

+    else:
+        logger.warning(
+            "Skipping configuring reactor metrics: unexpected reactor type: %r",
+            reactor,
+        )


This is the new warning when we see a new unexpected reactor type.

In my case, we we're running up against ProxiedReactor which isn't handled yet.

(o_O) What's a ProxiedReactor?! https://github.com/search?q=repo%3Atwisted%2Ftwisted%20ProxiedReactor&type=code

More investigation in #19385

This PR was mainly just to give better feedback regardless of the reactor.

devonh

Nice little addition to make it easier to track down missing metrics. 👍

MadLittleMods · 2026-01-15T21:49:51Z

Thanks for the review @devonh 🐢

…ker Complement tests (#19385) Follow-up to #19383 The [`ProxiedReactor`](https://github.com/element-hq/synapse/blob/079c52e16b3e75929380d59c5f1d85b66c1a4ccf/synapse/app/complement_fork_starter.py#L38-L71) is a special custom reactor used in the `synapse/app/complement_fork_starter.py`. It's used by default when using `WORKERS=1 ./scripts-dev/complement.sh` (see `SYNAPSE_USE_EXPERIMENTAL_FORKING_LAUNCHER`). The point of the forking launcher is to improve start-up times and the point of the [`ProxiedReactor`](https://github.com/element-hq/synapse/blob/079c52e16b3e75929380d59c5f1d85b66c1a4ccf/synapse/app/complement_fork_starter.py#L38-L71) is explained in the [docstring](https://github.com/element-hq/synapse/blob/079c52e16b3e75929380d59c5f1d85b66c1a4ccf/synapse/app/complement_fork_starter.py#L38-L56) (introduced in matrix-org/synapse#13127) ### Reproduction instructions 1. Using the Complement scripts **with workers**: `WORKERS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh ./tests/csapi` 1. `docker logs complement_csapi_dirty_hs1 2>&1 | grep -i "reactor"` 1. With these changes, notice `Twisted reactor: ProxiedReactor` but no warning about `Skipping configuring ReactorLastSeenMetric: unexpected reactor type: <__main__.ProxiedReactor object at 0x7fc0adaaea50>` 1. Cleanup: - `docker stop $(docker ps --all --filter "label=complement_context" --quiet)` - `docker rm $(docker ps --all --filter "label=complement_context" --quiet)` To test that we're actually seeing reactor metrics, I've been using this with the load-test runs in element-hq/synapse-rust-apps#397

This PR contains the following updates: | Package | Update | Change | |---|---|---| | [element-hq/synapse](https://github.com/element-hq/synapse) | minor | `1.145.0` → `1.146.0` | --- > ⚠️ **Warning** > > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>element-hq/synapse (element-hq/synapse)</summary> ### [`v1.146.0`](https://github.com/element-hq/synapse/releases/tag/v1.146.0) [Compare Source](element-hq/synapse@v1.145.0...v1.146.0rc1) ### Synapse 1.146.0 (2026-01-27) No significant changes since 1.146.0rc1. #### Deprecations and Removals - [MSC2697](matrix-org/matrix-spec-proposals#2697) (Dehydrated devices) has been removed, as the MSC is closed. Developers should migrate to [MSC3814](matrix-org/matrix-spec-proposals#3814). ([#19346](element-hq/synapse#19346)) - Support for Ubuntu 25.04 (Plucky Puffin) has been dropped. Synapse no longer builds debian packages for Ubuntu 25.04. ### Synapse 1.146.0rc1 (2026-01-20) #### Features - Add a new config option [`enable_local_media_storage`](https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#enable_local_media_storage) which controls whether media is additionally stored locally when using configured `media_storage_providers`. Setting this to `false` allows off-site media storage without a local cache. Contributed by Patrice Brend'amour [@dr](https://github.com/dr).allgood. ([#19204](element-hq/synapse#19204)) - Stabilise support for [MSC4312](matrix-org/matrix-spec-proposals#4312 `m.oauth` User-Interactive Auth stage for resetting cross-signing identity with the OAuth 2.0 API. The old, unstable name (`org.matrix.cross_signing_reset`) is now deprecated and will be removed in a future release. ([#19273](element-hq/synapse#19273)) - Refactor Grafana dashboard to use `server_name` label (instead of `instance`). ([#19337](element-hq/synapse#19337)) #### Bugfixes - Fix joining a restricted v12 room locally when no local room creator is present but local users with sufficient power levels are. Contributed by [@nexy7574](https://github.com/nexy7574). ([#19321](element-hq/synapse#19321)) - Fixed parallel calls to `/_matrix/media/v1/create` being ratelimited for appservices even if `rate_limited: false` was set in the registration. Contributed by [@tulir](https://github.com/tulir) @ Beeper. ([#19335](element-hq/synapse#19335)) - Fix a bug introduced in 1.61.0 where a user's membership in a room was accidentally ignored when considering access to historical state events in rooms with the "shared" history visibility. Contributed by Lukas Tautz. ([#19353](element-hq/synapse#19353)) - [MSC4140](matrix-org/matrix-spec-proposals#4140): Store the JSON content of scheduled delayed events as text instead of a byte array. This fixes the inability to schedule a delayed event with non-ASCII characters in its content. ([#19360](element-hq/synapse#19360)) - Always rollback database transactions when retrying (avoid orphaned connections). ([#19372](element-hq/synapse#19372)) - Fix `InFlightGauge` typing to allow upgrading to `prometheus_client` 0.24. ([#19379](element-hq/synapse#19379)) #### Updates to the Docker image - Add [Prometheus HTTP service discovery](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_sd_config) endpoint for easy discovery of all workers when using the `docker/Dockerfile-workers` image (see the [*Metrics* section of our Docker testing docs](docker/README-testing.md#metrics)). ([#19336](element-hq/synapse#19336)) #### Improved Documentation - Remove docs on legacy metric names (no longer in the codebase since 2022-12-06). ([#19341](element-hq/synapse#19341)) - Clarify how the estimated value of room complexity is calculated internally. ([#19384](element-hq/synapse#19384)) #### Internal Changes - Add an internal `cancel_task` API to the task scheduler. ([#19310](element-hq/synapse#19310)) - Tweak docstrings and signatures of `auth_types_for_event` and `get_catchup_room_event_ids`. ([#19320](element-hq/synapse#19320)) - Replace usage of deprecated `assertEquals` with `assertEqual` in unit test code. ([#19345](element-hq/synapse#19345)) - Drop support for Ubuntu 25.04 'Plucky Puffin', add support for Ubuntu 25.10 'Questing Quokka'. ([#19348](element-hq/synapse#19348)) - Revert "Add an Admin API endpoint for listing quarantined media ([#19268](element-hq/synapse#19268))". ([#19351](element-hq/synapse#19351)) - Bump `mdbook` from 0.4.17 to 0.5.2 and remove our custom table-of-contents plugin in favour of the new default functionality. ([#19356](element-hq/synapse#19356)) - Replace deprecated usage of PyGitHub's `GitRelease.title` with `.name` in release script. ([#19358](element-hq/synapse#19358)) - Update the Element logo in Synapse's README to be an absolute URL, allowing it to render on other sites (such as PyPI). ([#19368](element-hq/synapse#19368)) - Apply minor tweaks to v1.145.0 changelog. ([#19376](element-hq/synapse#19376)) - Update Grafana dashboard syntax to use the latest from importing/exporting with Grafana 12.3.1. ([#19381](element-hq/synapse#19381)) - Warn about skipping reactor metrics when using unknown reactor type. ([#19383](element-hq/synapse#19383)) - Add support for reactor metrics with the `ProxiedReactor` used in worker Complement tests. ([#19385](element-hq/synapse#19385)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://gitea.alexlebens.dev/alexlebens/infrastructure/pulls/3533 Co-authored-by: Renovate Bot <renovate-bot@alexlebens.net> Co-committed-by: Renovate Bot <renovate-bot@alexlebens.net>

Warn about skipping ReactorLastSeenMetric with unknown reactor type

923486a

MadLittleMods added the A-Metrics label Jan 15, 2026

MadLittleMods changed the title ~~Warn about skipping ReactorLastSeenMetric with unknown reactor type~~ Warn about skipping reactor metrics with unknown reactor type Jan 15, 2026

Be generic about "reactor metrics"

0cf610f

MadLittleMods changed the title ~~Warn about skipping reactor metrics with unknown reactor type~~ Warn about skipping reactor metrics when using unknown reactor type Jan 15, 2026

Add changelog

fe3d3e7

MadLittleMods commented Jan 15, 2026

View reviewed changes

MadLittleMods marked this pull request as ready for review January 15, 2026 20:03

MadLittleMods requested a review from a team as a code owner January 15, 2026 20:03

devonh approved these changes Jan 15, 2026

View reviewed changes

MadLittleMods merged commit 6363d77 into develop Jan 15, 2026
77 of 79 checks passed

MadLittleMods deleted the madlittlemods/warn-skip-ReactorLastSeenMetric-when-unknown-reactor-type branch January 15, 2026 21:49

MadLittleMods mentioned this pull request Jan 15, 2026

Add support for reactor metrics with the ProxiedReactor used in worker Complement tests #19385

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Warn about skipping reactor metrics when using unknown reactor type#19383

Warn about skipping reactor metrics when using unknown reactor type#19383
MadLittleMods merged 3 commits intodevelopfrom
madlittlemods/warn-skip-ReactorLastSeenMetric-when-unknown-reactor-type

MadLittleMods commented Jan 15, 2026 •

edited

Loading

Uh oh!

MadLittleMods Jan 15, 2026

Uh oh!

devonh Jan 15, 2026

Uh oh!

MadLittleMods Jan 15, 2026 •

edited

Loading

Uh oh!

clokep Jan 16, 2026

Uh oh!

MadLittleMods Jan 16, 2026

Uh oh!

devonh left a comment

Uh oh!

Uh oh!

MadLittleMods commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

MadLittleMods commented Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reproduction instructions

Pull Request Checklist

Uh oh!

MadLittleMods Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

devonh Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clokep Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

MadLittleMods Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

devonh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MadLittleMods commented Jan 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MadLittleMods commented Jan 15, 2026 •

edited

Loading

MadLittleMods Jan 15, 2026 •

edited

Loading