Refactor Grafana dashboard to use server_name label

[MadLittleMods]

Refactor Grafana dashboard to use server_name label:

• Update synapse_xxx (server-level) metrics to use server_name="$server_name", instead of instance="$instance"
• Add synapse_server_name_info metric to map Synapse server_names to the instances they're hosted on.
• For process level metrics, update to use xxx * on (instance, job, index) group_left(server_name) synapse_server_name_info{server_name="$server_name"}

All of the changes here are backwards compatible with whatever people were doing before with their Prometheus/Grafana dashboards.

Previously, the recommendation was to use the instance label to group everything under the same server:

synapse/docs/metrics-howto.md

Lines 93 to 147 in 803e4b4

	## Monitoring workers
	To monitor a Synapse installation using [workers](workers.md),
	every worker needs to be monitored independently, in addition to
	the main homeserver process. This is because workers don't send
	their metrics to the main homeserver process, but expose them
	directly (if they are configured to do so).
	To allow collecting metrics from a worker, you need to add a
	`metrics` listener to its configuration, by adding the following
	under `worker_listeners`:
	```yaml
	- type: metrics
	bind_address: ''
	port: 9101
	```
	The `bind_address` and `port` parameters should be set so that
	the resulting listener can be reached by prometheus, and they
	don't clash with an existing worker.
	With this example, the worker's metrics would then be available
	on `http://127.0.0.1:9101`.
	Example Prometheus target for Synapse with workers:
	```yaml
	- job_name: "synapse"
	scrape_interval: 15s
	metrics_path: "/_synapse/metrics"
	static_configs:
	- targets: ["my.server.here:port"]
	labels:
	instance: "my.server"
	job: "master"
	index: 1
	- targets: ["my.workerserver.here:port"]
	labels:
	instance: "my.server"
	job: "generic_worker"
	index: 1
	- targets: ["my.workerserver.here:port"]
	labels:
	instance: "my.server"
	job: "generic_worker"
	index: 2
	- targets: ["my.workerserver.here:port"]
	labels:
	instance: "my.server"
	job: "media_repository"
	index: 1
	```
	Labels (`instance`, `job`, `index`) can be defined as anything.
	The labels are used to group graphs in grafana.

But the instance label actually has a special meaning and we're actually abusing it by using it that way:

instance: The <host>:<port> part of the target's URL that was scraped.

-- https://prometheus.io/docs/concepts/jobs_instances/#automatically-generated-labels-and-time-series

Since #18592 (Synapse v1.139.0), we now have the server_name label to use instead.

---

Additionally, the assumption that a single process is serving a single server is no longer true with Synapse Pro for small hosts.

Part of https://github.com/element-hq/synapse-small-hosts/issues/106

Motivating use case

Although this change also benefits Synapse Pro for small hosts (https://github.com/element-hq/synapse-small-hosts/issues/106), this is actually spawning from adding Prometheus metrics to our workerized Docker image (#19324, #19336) with a more correct label setup (without instance) and wanting the dashboard to be better.

Testing strategy

1. Make sure your firewall allows the Docker containers to communicate to the host (host.docker.internal) so they can access exposed ports of other Docker containers. We want to allow Synapse to access the Prometheus container and Grafana to access to the Prometheus container.
   • sudo ufw allow in on docker0 comment "Allow traffic from the default Docker network to the host machine (host.docker.internal)"
   • sudo ufw allow in on br-+ comment "(from Matrix Complement testing) Allow traffic from custom Docker networks to the host machine (host.docker.internal)"
   • Complement firewall docs
2. Build the Docker image for Synapse: docker build -t matrixdotorg/synapse -f docker/Dockerfile . (docs)
3. Generate config for Synapse:

   docker run -it --rm \
       --mount type=volume,src=synapse-data,dst=/data \
       -e SYNAPSE_SERVER_NAME=my.docker.synapse.server \
       -e SYNAPSE_REPORT_STATS=yes \
       -e SYNAPSE_ENABLE_METRICS=1 \
       matrixdotorg/synapse:latest generate
   

4. Start Synapse:

   docker run -d --name synapse \
      --mount type=volume,src=synapse-data,dst=/data \
      -p 8008:8008 \
      -p 19090:19090 \
      matrixdotorg/synapse:latest
   

5. You should be able to see metrics from Synapse at http://localhost:19090/_synapse/metrics
6. Create a Prometheus config (prometheus.yml)

   global:
     scrape_interval: 15s
     scrape_timeout: 15s
     evaluation_interval: 15s
   
   scrape_configs:
     - job_name: prometheus
       scrape_interval: 15s
       metrics_path: /_synapse/metrics
       scheme: http
       static_configs:
         - targets:
             # This should point to the Synapse metrics listener (we're using `host.docker.internal` because this is from within the Prometheus container)
             - host.docker.internal:19090

7. Start Prometheus (update the volume bind mount to the config you just saved somewhere):

   docker run \
       --detach \
       --name=prometheus \
       --add-host host.docker.internal:host-gateway \
       -p 9090:9090 \
       -v ~/Documents/code/random/prometheus-config/prometheus.yml:/etc/prometheus/prometheus.yml \
       prom/prometheus
   

8. Make sure you're seeing some data in Prometheus. On http://localhost:9090/query, search for synapse_build_info
9. Start Grafana

   docker run -d --name=grafana --add-host host.docker.internal:host-gateway -p 3000:3000 grafana/grafana
   

10. Visit the Grafana dashboard, http://localhost:3000/ (Credentials: admin/admin)
11. Connections -> Data Sources -> Add data source -> Prometheus
    • Prometheus server URL: http://host.docker.internal:9090
12. Import the Synapse dashboard: contrib/grafana/synapse.json

To test workers, you can use the testing strategy from #19336 (assumes both changes from this PR and the other PR are combined)

Dev notes

How to stress the deploys annotation:

1. docker exec -it synapse /bin/bash
2. vim /usr/local/lib/python3.13/site-packages/synapse/util/__init__.py
3. Edit SYNAPSE_VERSION
4. docker stop synapse
5. docker start synapse

Todo

• Add server_name variable to dashboard
• Figure out how to better display the process_ metrics
• Update deploys annotation

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct (run the linters)

[MadLittleMods]

Updating all of the synapse_xxx server-level metrics to use $server_name instead of $instance

[MadLittleMods]

No longer needed as the recommendation is to rely on the server_name label

[MadLittleMods]

Updating all of the process-level metrics with this pattern:

xxx * on(instance, job, index) group_left(server_name)
synapse_server_name_to_instance_mapping{server_name="$server_name"}

With instance_to_server_name_mapping, looking up the instance that the $server_name lives to match against the process-level metric.

[MadLittleMods]

Skip reviewing the annotation query until the end (because it's the most complicated change)

It's the same basic pattern upgrade as the other process-level metrics but we just have an extra * on ... layer

[reivilibre]

But the instance label actually has a special meaning and we're actually abusing it by using it that way:

instance: The : part of the target's URL that was scraped.

-- prometheus.io/docs/concepts/jobs_instances#automatically-generated-labels-and-time-series

Not really material to this actual PR, but just so you know I think that's just a 'sane default' but in my experience it's quite conventional to override it when you know better, e.g. using the blackbox exporter you typically relabel instance to be something more meaningful. (e.g. target_label: instance on https://prometheus.io/docs/guides/multi-target-exporter/)

[MadLittleMods]

Thanks for the sharing the link and context!

I think that overriding instance to mean something else is a mistake, with vhosting being the shining example of how this falls apart otherwise. I understand how it can work out fine in simple cases though. It's all conventions anyway.

The 'sane default' outlook on this makes sense as that is what happens by default 🤔 - With the way I'm thinking about this, a more specific definition might be "URL of the server process that the metrics came from". This covers all of the complicated scenarios:

1. One server per process -> instance points to the process
2. Multiple servers per process (vhosting) with one metrics endpoint for the process -> instance points to the whole process
3. Proxying metrics from other servers -> instance points to where the metrics came from

There is one layer of complexity that even this metrics setup doesn't handle yet. For example with Synapse Pro for small hosts, if we instead wanted to pack in multiple workers (for the same homeserver) into the same process (instead of multiple monolith homeservers), we wouldn't have the necessary labels to differentiate them. In addition to server_name, we would additionally need to label all of the server-level metrics with worker_name (or job/index). (I understand this use case doesn't make sense but it might for another person's setup)

I feel like a lot of docs are missing around how to handle metrics with vhosting. I would definitely prefer to lean on conventions but there doesn't seem to be any.

[reivilibre]

still need to read the big JSON dump :-))

[reivilibre]

so if I'm getting this right, it's not really a map as such (the value is meaningless), just that it's a dummy metric we can rely on to always be there that will always have {instance, server_name} levels.

It seems this type of metric (labels carry data for the length of the process, value is fixed at 1) conventionally has a _info suffix, e.g. see https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#info

Would it make more sense to just call this synapse_server_names_info or something of that ilk.

I suppose in a vhosting model, when virtual-hosts are deregistered, we would remove their label from here?

[MadLittleMods]

so if I'm getting this right, it's not really a map as such (the value is meaningless), just that it's a dummy metric we can rely on to always be there that will always have {instance, server_name} levels.

I think you have the correct grasp on it. The labels do allow us to map server_name -> instance though (and "mapping" is the purpose of it) 🤷

It seems this type of metric (labels carry data for the length of the process, value is fixed at 1) conventionally has a _info suffix, e.g. see https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#info

Would it make more sense to just call this synapse_server_names_info or something of that ilk.

Sounds like a good practice to follow 👍

I suppose in a vhosting model, when virtual-hosts are deregistered, we would remove their label from here?

Yes. My current thinking is to remove it when we hs.shutdown() but I think this is better as a follow-up where we can describe it directly and test it.

Probably the final piece for https://github.com/element-hq/synapse-small-hosts/issues/106

[MadLittleMods]

Would it make more sense to just call this synapse_server_names_info or something of that ilk.

Renamed to synapse_server_name_info 👍

[reivilibre]

quite an interesting (if a little boilerplatey) approach, multiplying by some other constant-1 metric to match on more labels.

I think I'm getting it now, I'm happy with this.

I leave the decision around naming / following the convention or not up to you — I don't suspect it matters an awful lot and I don't know if this metric feels like a 'conventional _info metric' beyond matching the basic pattern, or not.

[MadLittleMods]

Thanks for the review @reivilibre 🐗