Memory Leak because one client sends 20Hz requests to POST `client/v3/keys/upload`. One time key signed_curve25519:AAAAAAAAAFA already exists

[leofah]

Description

A few days ago the memory of our synapse instance started leaking. Ca 100MB/hour and 400MB/hour.
With the grafana dashboard we found a unusually high request rate (20Hz-100Hz) to POST client/v3/keys/upload.
The client tries to upload a one_time_key, but fails as a key with the same key_id already exists.
However the client does not stop sending the requests.
Of course the client should not send that many requests, but this should not lead to a memory leak on the server.

The client recently updated from element 1.12.7 to 1.12.8. With 1.12.7 the client sent the same request, but only every 30s. With 1.12.8 the client started sending the request multiple times per second, leading to the memory leak.
One can see in the dashboard, memory only leaks, when there are that many requests.
Probably memory still leaks, during a low request rate, but then the leak is not noticeable.

We upgraded synapse from 1.142.0 to the newest version 1.145.0, but it did not resolve the leak.

Steps to reproduce

• memory leaks as long as the "malicious" client sends requests

Homeserver

homeserver of our university

Synapse Version

1.145.0

Installation Method

Debian packages from packages.matrix.org

Database

Postgresql 16 on same machine. Upgraded from 12 to 14 to 16, when upgrading ubuntu from 20.04 to 22.04 to 24.04

Workers

Single process

Platform

Ubuntu 24.04.
VMware virtual machine.

Configuration

The garbage collection settings are set to default.

Relevant log output

2026-01-19 12:34:51,001 - synapse.http.server - 131 - INFO - POST-5688809 - <XForwardedForRequest at 0x7ec9ed2bca10 method='POST' uri='/_matrix/client/v3/keys/upload' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - One time key signed_curve25519:AAAAAAAAAFA already exists. Old key: {"key":"9lb05QrEBh7mV9PVSD1eBzXQjt4dEAOaYjlmKgwr8kI","signatures":{"[user_redacted]":{"ed25519:RWDSUMBAID":"QCgG8Jj/8D/SeX3zuh+Fx3griM7ykurmJN30ypodMjhLks0tlDzEsBXraBJeaJYwInOpL7WD5/2wn1oCO7pABQ"}}}; new key: {'key': 'daYxx2uda7KtVrKtxAKanhJ7ip9yzvFEjJzT7Qg0/x0', 'signatures': {'[user_redacted]': {'ed25519:RWDSUMBAID': 'GT9G1g5NTwWPnQGRmf5JcSddW9ZQvHA3Ly3+zs04gnQ4d9OKX5MrbcZ33o27Xd2r43ebZ1a3bDJ/IfUmfH58Cg'}}}
2026-01-19 12:34:51,001 - synapse.access.http.8008 - 641 - INFO - POST-5688809 - [client_ip] - 8008 - {[user_redacted]} Processed request: 0.003sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.002sec/3) 528B 400 "POST /_matrix/client/v3/keys/upload HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.8 Chrome/142.0.7444.226 Electron/39.2.6 Safari/537.36" [0 dbevts]
2026-01-19 12:34:51,016 - synapse.handlers.e2e_keys - 955 - INFO - POST-5688810 - Adding one_time_keys dict_keys(['signed_curve25519:AAAAAAAAAE0', 'signed_curve25519:AAAAAAAAAE4', 'signed_curve25519:AAAAAAAAAE8', 'signed_curve25519:AAAAAAAAAEc', 'signed_curve25519:AAAAAAAAAEg', 'signed_curve25519:AAAAAAAAAEk', 'signed_curve25519:AAAAAAAAAEo', 'signed_curve25519:AAAAAAAAAEs', 'signed_curve25519:AAAAAAAAAEw', 'signed_curve25519:AAAAAAAAAFA', 'signed_curve25519:AAAAAAAAAFE', 'signed_curve25519:AAAAAAAAAFI', 'signed_curve25519:AAAAAAAAAFM', 'signed_curve25519:AAAAAAAAAFQ', 'signed_curve25519:AAAAAAAAAFU', 'signed_curve25519:AAAAAAAAAFY', 'signed_curve25519:AAAAAAAAAFc', 'signed_curve25519:AAAAAAAAAFg', 'signed_curve25519:AAAAAAAAAFk']) for device 'RWDSUMBAID' for user '[user_redacted]' at 1768822491008
2026-01-19 12:34:51,022 - synapse.http.server - 131 - INFO - POST-5688810 - <XForwardedForRequest at 0x7ec9ed154fe0 method='POST' uri='/_matrix/client/v3/keys/upload' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - One time key signed_curve25519:AAAAAAAAAFA already exists. Old key: {"key":"9lb05QrEBh7mV9PVSD1eBzXQjt4dEAOaYjlmKgwr8kI","signatures":{"[user_redacted]":{"ed25519:RWDSUMBAID":"QCgG8Jj/8D/SeX3zuh+Fx3griM7ykurmJN30ypodMjhLks0tlDzEsBXraBJeaJYwInOpL7WD5/2wn1oCO7pABQ"}}}; new key: {'key': 'daYxx2uda7KtVrKtxAKanhJ7ip9yzvFEjJzT7Qg0/x0', 'signatures': {'[user_redacted]': {'ed25519:RWDSUMBAID': 'GT9G1g5NTwWPnQGRmf5JcSddW9ZQvHA3Ly3+zs04gnQ4d9OKX5MrbcZ33o27Xd2r43ebZ1a3bDJ/IfUmfH58Cg'}}}
2026-01-19 12:34:51,022 - synapse.access.http.8008 - 641 - INFO - POST-5688810 - [client_ip] - 8008 - {[user_redacted]} Processed request: 0.014sec/0.000sec (0.001sec, 0.000sec) (0.000sec/0.013sec/3) 528B 400 "POST /_matrix/client/v3/keys/upload HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.8 Chrome/142.0.7444.226 Electron/39.2.6 Safari/537.36" [0 dbevts]
2026-01-19 12:34:51,034 - synapse.handlers.e2e_keys - 955 - INFO - POST-5688811 - Adding one_time_keys dict_keys(['signed_curve25519:AAAAAAAAAE0', 'signed_curve25519:AAAAAAAAAE4', 'signed_curve25519:AAAAAAAAAE8', 'signed_curve25519:AAAAAAAAAEc', 'signed_curve25519:AAAAAAAAAEg', 'signed_curve25519:AAAAAAAAAEk', 'signed_curve25519:AAAAAAAAAEo', 'signed_curve25519:AAAAAAAAAEs', 'signed_curve25519:AAAAAAAAAEw', 'signed_curve25519:AAAAAAAAAFA', 'signed_curve25519:AAAAAAAAAFE', 'signed_curve25519:AAAAAAAAAFI', 'signed_curve25519:AAAAAAAAAFM', 'signed_curve25519:AAAAAAAAAFQ', 'signed_curve25519:AAAAAAAAAFU', 'signed_curve25519:AAAAAAAAAFY', 'signed_curve25519:AAAAAAAAAFc', 'signed_curve25519:AAAAAAAAAFg', 'signed_curve25519:AAAAAAAAAFk']) for device 'RWDSUMBAID' for user '[user_redacted]' at 1768822491032
2026-01-19 12:34:51,034 - twisted - 273 - INFO - POST-5688811 - 2026-01-19 12:34:50,988 - synapse.handlers.e2e_keys - 955 - INFO - POST-5688808 - Adding one_time_keys dict_keys(['signed_curve25519:AAAAAAAAAE0', 'signed_curve25519:AAAAAAAAAE4', 'signed_curve25519:AAAAAAAAAE8', 'signed_curve25519:AAAAAAAAAEc', 'signed_curve25519:AAAAAAAAAEg', 'signed_curve25519:AAAAAAAAAEk', 'signed_curve25519:AAAAAAAAAEo', 'signed_curve25519:AAAAAAAAAEs', 'signed_curve25519:AAAAAAAAAEw', 'signed_curve25519:AAAAAAAAAFA', 'signed_curve25519:AAAAAAAAAFE', 'signed_curve25519:AAAAAAAAAFI', 'signed_curve25519:AAAAAAAAAFM', 'signed_curve25519:AAAAAAAAAFQ', 'signed_curve25519:AAAAAAAAAFU', 'signed_curve25519:AAAAAAAAAFY', 'signed_curve25519:AAAAAAAAAFc', 'signed_curve25519:AAAAAAAAAFg', 'signed_curve25519:AAAAAAAAAFk']) for device 'RWDSUMBAID' for user '[user_redacted]' at 1768822490987
2026-01-19 12:34:51,036 - synapse.http.server - 131 - INFO - POST-5688811 - <XForwardedForRequest at 0x7ec9ec9508f0 method='POST' uri='/_matrix/client/v3/keys/upload' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - One time key signed_curve25519:AAAAAAAAAFA already exists. Old key: {"key":"9lb05QrEBh7mV9PVSD1eBzXQjt4dEAOaYjlmKgwr8kI","signatures":{"[user_redacted]":{"ed25519:RWDSUMBAID":"QCgG8Jj/8D/SeX3zuh+Fx3griM7ykurmJN30ypodMjhLks0tlDzEsBXraBJeaJYwInOpL7WD5/2wn1oCO7pABQ"}}}; new key: {'key': 'daYxx2uda7KtVrKtxAKanhJ7ip9yzvFEjJzT7Qg0/x0', 'signatures': {'[user_redacted]': {'ed25519:RWDSUMBAID': 'GT9G1g5NTwWPnQGRmf5JcSddW9ZQvHA3Ly3+zs04gnQ4d9OKX5MrbcZ33o27Xd2r43ebZ1a3bDJ/IfUmfH58Cg'}}}
2026-01-19 12:34:51,036 - synapse.access.http.8008 - 641 - INFO - POST-5688811 - [client_ip] - 8008 - {[user_redacted]} Processed request: 0.004sec/0.000sec (0.002sec, 0.000sec) (0.000sec/0.002sec/3) 528B 400 "POST /_matrix/client/v3/keys/upload HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.8 Chrome/142.0.7444.226 Electron/39.2.6 Safari/537.36" [0 dbevts]
2026-01-19 12:34:51,046 - synapse.handlers.e2e_keys - 955 - INFO - POST-5688812 - Adding one_time_keys dict_keys(['signed_curve25519:AAAAAAAAAE0', 'signed_curve25519:AAAAAAAAAE4', 'signed_curve25519:AAAAAAAAAE8', 'signed_curve25519:AAAAAAAAAEc', 'signed_curve25519:AAAAAAAAAEg', 'signed_curve25519:AAAAAAAAAEk', 'signed_curve25519:AAAAAAAAAEo', 'signed_curve25519:AAAAAAAAAEs', 'signed_curve25519:AAAAAAAAAEw', 'signed_curve25519:AAAAAAAAAFA', 'signed_curve25519:AAAAAAAAAFE', 'signed_curve25519:AAAAAAAAAFI', 'signed_curve25519:AAAAAAAAAFM', 'signed_curve25519:AAAAAAAAAFQ', 'signed_curve25519:AAAAAAAAAFU', 'signed_curve25519:AAAAAAAAAFY', 'signed_curve25519:AAAAAAAAAFc', 'signed_curve25519:AAAAAAAAAFg', 'signed_curve25519:AAAAAAAAAFk']) for device 'RWDSUMBAID' for user '[user_redacted]' at 1768822491045
2026-01-19 12:34:51,048 - synapse.http.server - 131 - INFO - POST-5688812 - <XForwardedForRequest at 0x7ec9ed6f63c0 method='POST' uri='/_matrix/client/v3/keys/upload' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - One time key signed_curve25519:AAAAAAAAAFA already exists. Old key: {"key":"9lb05QrEBh7mV9PVSD1eBzXQjt4dEAOaYjlmKgwr8kI","signatures":{"[user_redacted]":{"ed25519:RWDSUMBAID":"QCgG8Jj/8D/SeX3zuh+Fx3griM7ykurmJN30ypodMjhLks0tlDzEsBXraBJeaJYwInOpL7WD5/2wn1oCO7pABQ"}}}; new key: {'key': 'daYxx2uda7KtVrKtxAKanhJ7ip9yzvFEjJzT7Qg0/x0', 'signatures': {'[user_redacted]': {'ed25519:RWDSUMBAID': 'GT9G1g5NTwWPnQGRmf5JcSddW9ZQvHA3Ly3+zs04gnQ4d9OKX5MrbcZ33o27Xd2r43ebZ1a3bDJ/IfUmfH58Cg'}}}
2026-01-19 12:34:51,048 - synapse.access.http.8008 - 641 - INFO - POST-5688812 - [client_ip] - 8008 - {[user_redacted]} Processed request: 0.003sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.002sec/3) 528B 400 "POST /_matrix/client/v3/keys/upload HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.8 Chrome/142.0.7444.226 Electron/39.2.6 Safari/537.36" [0 dbevts]
2026-01-19 12:34:51,059 - synapse.handlers.e2e_keys - 955 - INFO - POST-5688813 - Adding one_time_keys dict_keys(['signed_curve25519:AAAAAAAAAE0', 'signed_curve25519:AAAAAAAAAE4', 'signed_curve25519:AAAAAAAAAE8', 'signed_curve25519:AAAAAAAAAEc', 'signed_curve25519:AAAAAAAAAEg', 'signed_curve25519:AAAAAAAAAEk', 'signed_curve25519:AAAAAAAAAEo', 'signed_curve25519:AAAAAAAAAEs', 'signed_curve25519:AAAAAAAAAEw', 'signed_curve25519:AAAAAAAAAFA', 'signed_curve25519:AAAAAAAAAFE', 'signed_curve25519:AAAAAAAAAFI', 'signed_curve25519:AAAAAAAAAFM', 'signed_curve25519:AAAAAAAAAFQ', 'signed_curve25519:AAAAAAAAAFU', 'signed_curve25519:AAAAAAAAAFY', 'signed_curve25519:AAAAAAAAAFc', 'signed_curve25519:AAAAAAAAAFg', 'signed_curve25519:AAAAAAAAAFk']) for device 'RWDSUMBAID' for user '[user_redacted]' at 1768822491058
2026-01-19 12:34:51,061 - synapse.http.server - 131 - INFO - POST-5688813 - <XForwardedForRequest at 0x7ec9ec7c0500 method='POST' uri='/_matrix/client/v3/keys/upload' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - One time key signed_curve25519:AAAAAAAAAFA already exists. Old key: {"key":"9lb05QrEBh7mV9PVSD1eBzXQjt4dEAOaYjlmKgwr8kI","signatures":{"[user_redacted]":{"ed25519:RWDSUMBAID":"QCgG8Jj/8D/SeX3zuh+Fx3griM7ykurmJN30ypodMjhLks0tlDzEsBXraBJeaJYwInOpL7WD5/2wn1oCO7pABQ"}}}; new key: {'key': 'daYxx2uda7KtVrKtxAKanhJ7ip9yzvFEjJzT7Qg0/x0', 'signatures': {'[user_redacted]': {'ed25519:RWDSUMBAID': 'GT9G1g5NTwWPnQGRmf5JcSddW9ZQvHA3Ly3+zs04gnQ4d9OKX5MrbcZ33o27Xd2r43ebZ1a3bDJ/IfUmfH58Cg'}}}
2026-01-19 12:34:51,061 - synapse.access.http.8008 - 641 - INFO - POST-5688813 - [client_ip] - 8008 - {[user_redacted]} Processed request: 0.003sec/0.000sec (0.001sec, 0.000sec) (0.000sec/0.002sec/3) 528B 400 "POST /_matrix/client/v3/keys/upload HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.8 Chrome/142.0.7444.226 Electron/39.2.6 Safari/537.36" [0 dbevts]
2026-01-19 12:34:51,072 - synapse.handlers.e2e_keys - 955 - INFO - POST-5688814 - Adding one_time_keys dict_keys(['signed_curve25519:AAAAAAAAAE0', 'signed_curve25519:AAAAAAAAAE4', 'signed_curve25519:AAAAAAAAAE8', 'signed_curve25519:AAAAAAAAAEc', 'signed_curve25519:AAAAAAAAAEg', 'signed_curve25519:AAAAAAAAAEk', 'signed_curve25519:AAAAAAAAAEo', 'signed_curve25519:AAAAAAAAAEs', 'signed_curve25519:AAAAAAAAAEw', 'signed_curve25519:AAAAAAAAAFA', 'signed_curve25519:AAAAAAAAAFE', 'signed_curve25519:AAAAAAAAAFI', 'signed_curve25519:AAAAAAAAAFM', 'signed_curve25519:AAAAAAAAAFQ', 'signed_curve25519:AAAAAAAAAFU', 'signed_curve25519:AAAAAAAAAFY', 'signed_curve25519:AAAAAAAAAFc', 'signed_curve25519:AAAAAAAAAFg', 'signed_curve25519:AAAAAAAAAFk']) for device 'RWDSUMBAID' for user '[user_redacted]' at 1768822491071
2026-01-19 12:34:51,074 - synapse.http.server - 131 - INFO - POST-5688814 - <XForwardedForRequest at 0x7ec9ec3d9a00 method='POST' uri='/_matrix/client/v3/keys/upload' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - One time key signed_curve25519:AAAAAAAAAFA already exists. Old key: {"key":"9lb05QrEBh7mV9PVSD1eBzXQjt4dEAOaYjlmKgwr8kI","signatures":{"[user_redacted]":{"ed25519:RWDSUMBAID":"QCgG8Jj/8D/SeX3zuh+Fx3griM7ykurmJN30ypodMjhLks0tlDzEsBXraBJeaJYwInOpL7WD5/2wn1oCO7pABQ"}}}; new key: {'key': 'daYxx2uda7KtVrKtxAKanhJ7ip9yzvFEjJzT7Qg0/x0', 'signatures': {'[user_redacted]': {'ed25519:RWDSUMBAID': 'GT9G1g5NTwWPnQGRmf5JcSddW9ZQvHA3Ly3+zs04gnQ4d9OKX5MrbcZ33o27Xd2r43ebZ1a3bDJ/IfUmfH58Cg'}}}
2026-01-19 12:34:51,074 - twisted - 273 - INFO - POST-5688814 - 2026-01-19 12:34:51,036 - synapse.http.server - 131 - INFO - POST-5688811 - <XForwardedForRequest at 0x7ec9ec9508f0 method='POST' uri='/_matrix/client/v3/keys/upload' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - One time key signed_curve25519:AAAAAAAAAFA already exists. Old key: {"key":"9lb05QrEBh7mV9PVSD1eBzXQjt4dEAOaYjlmKgwr8kI","signatures":{"[user_redacted]":{"ed25519:RWDSUMBAID":"QCgG8Jj/8D/SeX3zuh+Fx3griM7ykurmJN30ypodMjhLks0tlDzEsBXraBJeaJYwInOpL7WD5/2wn1oCO7pABQ"}}}; new key: {'key': 'daYxx2uda7KtVrKtxAKanhJ7ip9yzvFEjJzT7Qg0/x0', 'signatures': {'[user_redacted]': {'ed25519:RWDSUMBAID': 'GT9G1g5NTwWPnQGRmf5JcSddW9ZQvHA3Ly3+zs04gnQ4d9OKX5MrbcZ33o27Xd2r43ebZ1a3bDJ/IfUmfH58Cg'}}}


And with element 1.12.7 the same request every 30s


2026-01-11 00:02:01,216 - synapse.access.http.8008 - 521 - INFO - GET-9159943 - [client_ip] - 8008 - {[user_redacted]} Processed request: 30.002sec/0.000sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 202B 200 "GET /_matrix/client/v3/sync?filter=2&timeout=30000&org.matrix.msc4222.use_state_after=true&set_presence=unavailable&since=s9782324_78224410_96728_16697952_5795792_2669283_4722665_13781625_0_1826_2 HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.7 Chrome/142.0.7444.175 Electron/39.2.3 Safari/537.36" [0 dbevts]
2026-01-11 00:02:01,291 - synapse.handlers.e2e_keys - 954 - INFO - POST-9160055 - Adding one_time_keys dict_keys(['signed_curve25519:AAAAAAAAAE0', 'signed_curve25519:AAAAAAAAAE4', 'signed_curve25519:AAAAAAAAAE8', 'signed_curve25519:AAAAAAAAAEc', 'signed_curve25519:AAAAAAAAAEg', 'signed_curve25519:AAAAAAAAAEk', 'signed_curve25519:AAAAAAAAAEo', 'signed_curve25519:AAAAAAAAAEs', 'signed_curve25519:AAAAAAAAAEw', 'signed_curve25519:AAAAAAAAAFA', 'signed_curve25519:AAAAAAAAAFE', 'signed_curve25519:AAAAAAAAAFI', 'signed_curve25519:AAAAAAAAAFM', 'signed_curve25519:AAAAAAAAAFQ', 'signed_curve25519:AAAAAAAAAFU', 'signed_curve25519:AAAAAAAAAFY', 'signed_curve25519:AAAAAAAAAFc', 'signed_curve25519:AAAAAAAAAFg', 'signed_curve25519:AAAAAAAAAFk']) for device 'RWDSUMBAID' for user '[user_redacted]' at 1768086121289
2026-01-11 00:02:01,294 - synapse.http.server - 132 - INFO - POST-9160055 - <XForwardedForRequest at 0x7b965bfd7cb0 method='POST' uri='/_matrix/client/v3/keys/upload' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - One time key signed_curve25519:AAAAAAAAAFA already exists. Old key: {"key":"9lb05QrEBh7mV9PVSD1eBzXQjt4dEAOaYjlmKgwr8kI","signatures":{"[user_redacted]":{"ed25519:RWDSUMBAID":"QCgG8Jj/8D/SeX3zuh+Fx3griM7ykurmJN30ypodMjhLks0tlDzEsBXraBJeaJYwInOpL7WD5/2wn1oCO7pABQ"}}}; new key: {'key': 'daYxx2uda7KtVrKtxAKanhJ7ip9yzvFEjJzT7Qg0/x0', 'signatures': {'[user_redacted]': {'ed25519:RWDSUMBAID': 'GT9G1g5NTwWPnQGRmf5JcSddW9ZQvHA3Ly3+zs04gnQ4d9OKX5MrbcZ33o27Xd2r43ebZ1a3bDJ/IfUmfH58Cg'}}}
2026-01-11 00:02:01,294 - synapse.access.http.8008 - 521 - INFO - POST-9160055 - [client_ip] - 8008 - {[user_redacted]} Processed request: 0.006sec/0.000sec (0.002sec, 0.000sec) (0.000sec/0.004sec/3) 528B 400 "POST /_matrix/client/v3/keys/upload HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.7 Chrome/142.0.7444.175 Electron/39.2.3 Safari/537.36" [0 dbevts]
2026-01-11 00:02:31,283 - synapse.access.http.8008 - 521 - INFO - GET-9160054 - [client_ip] - 8008 - {[user_redacted]} Processed request: 30.002sec/0.000sec (0.002sec, 0.000sec) (0.000sec/0.000sec/0) 202B 200 "GET /_matrix/client/v3/sync?filter=2&timeout=30000&org.matrix.msc4222.use_state_after=true&set_presence=unavailable&since=s9782324_78224410_96728_16697952_5795792_2669283_4722665_13781626_0_1826_2 HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.7 Chrome/142.0.7444.175 Electron/39.2.3 Safari/537.36" [0 dbevts]
2026-01-11 00:02:31,359 - synapse.handlers.e2e_keys - 954 - INFO - POST-9160179 - Adding one_time_keys dict_keys(['signed_curve25519:AAAAAAAAAE0', 'signed_curve25519:AAAAAAAAAE4', 'signed_curve25519:AAAAAAAAAE8', 'signed_curve25519:AAAAAAAAAEc', 'signed_curve25519:AAAAAAAAAEg', 'signed_curve25519:AAAAAAAAAEk', 'signed_curve25519:AAAAAAAAAEo', 'signed_curve25519:AAAAAAAAAEs', 'signed_curve25519:AAAAAAAAAEw', 'signed_curve25519:AAAAAAAAAFA', 'signed_curve25519:AAAAAAAAAFE', 'signed_curve25519:AAAAAAAAAFI', 'signed_curve25519:AAAAAAAAAFM', 'signed_curve25519:AAAAAAAAAFQ', 'signed_curve25519:AAAAAAAAAFU', 'signed_curve25519:AAAAAAAAAFY', 'signed_curve25519:AAAAAAAAAFc', 'signed_curve25519:AAAAAAAAAFg', 'signed_curve25519:AAAAAAAAAFk']) for device 'RWDSUMBAID' for user '[user_redacted]' at 1768086151354
2026-01-11 00:02:31,363 - synapse.http.server - 132 - INFO - POST-9160179 - <XForwardedForRequest at 0x7b965bfd79b0 method='POST' uri='/_matrix/client/v3/keys/upload' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - One time key signed_curve25519:AAAAAAAAAFA already exists. Old key: {"key":"9lb05QrEBh7mV9PVSD1eBzXQjt4dEAOaYjlmKgwr8kI","signatures":{"[user_redacted]":{"ed25519:RWDSUMBAID":"QCgG8Jj/8D/SeX3zuh+Fx3griM7ykurmJN30ypodMjhLks0tlDzEsBXraBJeaJYwInOpL7WD5/2wn1oCO7pABQ"}}}; new key: {'key': 'daYxx2uda7KtVrKtxAKanhJ7ip9yzvFEjJzT7Qg0/x0', 'signatures': {'[user_redacted]': {'ed25519:RWDSUMBAID': 'GT9G1g5NTwWPnQGRmf5JcSddW9ZQvHA3Ly3+zs04gnQ4d9OKX5MrbcZ33o27Xd2r43ebZ1a3bDJ/IfUmfH58Cg'}}}
2026-01-11 00:02:31,363 - synapse.access.http.8008 - 521 - INFO - POST-9160179 - [client_ip] - 8008 - {[user_redacted]} Processed request: 0.010sec/0.000sec (0.001sec, 0.000sec) (0.001sec/0.008sec/3) 528B 400 "POST /_matrix/client/v3/keys/upload HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.7 Chrome/142.0.7444.175 Electron/39.2.3 Safari/537.36" [0 dbevts]
2026-01-11 00:03:01,349 - synapse.access.http.8008 - 521 - INFO - GET-9160178 - [client_ip] - 8008 - {[user_redacted]} Processed request: 30.001sec/0.000sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 202B 200 "GET /_matrix/client/v3/sync?filter=2&timeout=30000&org.matrix.msc4222.use_state_after=true&set_presence=unavailable&since=s9782324_78224410_96728_16697952_5795792_2669283_4722665_13781626_0_1826_2 HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.7 Chrome/142.0.7444.175 Electron/39.2.3 Safari/537.36" [0 dbevts]
2026-01-11 00:03:01,427 - synapse.handlers.e2e_keys - 954 - INFO - POST-9160297 - Adding one_time_keys dict_keys(['signed_curve25519:AAAAAAAAAE0', 'signed_curve25519:AAAAAAAAAE4', 'signed_curve25519:AAAAAAAAAE8', 'signed_curve25519:AAAAAAAAAEc', 'signed_curve25519:AAAAAAAAAEg', 'signed_curve25519:AAAAAAAAAEk', 'signed_curve25519:AAAAAAAAAEo', 'signed_curve25519:AAAAAAAAAEs', 'signed_curve25519:AAAAAAAAAEw', 'signed_curve25519:AAAAAAAAAFA', 'signed_curve25519:AAAAAAAAAFE', 'signed_curve25519:AAAAAAAAAFI', 'signed_curve25519:AAAAAAAAAFM', 'signed_curve25519:AAAAAAAAAFQ', 'signed_curve25519:AAAAAAAAAFU', 'signed_curve25519:AAAAAAAAAFY', 'signed_curve25519:AAAAAAAAAFc', 'signed_curve25519:AAAAAAAAAFg', 'signed_curve25519:AAAAAAAAAFk']) for device 'RWDSUMBAID' for user '[user_redacted]' at 1768086181426
2026-01-11 00:03:01,429 - synapse.http.server - 132 - INFO - POST-9160297 - <XForwardedForRequest at 0x7b965ea62cf0 method='POST' uri='/_matrix/client/v3/keys/upload' clientproto='HTTP/1.0' site='8008'> SynapseError: 400 - One time key signed_curve25519:AAAAAAAAAFA already exists. Old key: {"key":"9lb05QrEBh7mV9PVSD1eBzXQjt4dEAOaYjlmKgwr8kI","signatures":{"[user_redacted]":{"ed25519:RWDSUMBAID":"QCgG8Jj/8D/SeX3zuh+Fx3griM7ykurmJN30ypodMjhLks0tlDzEsBXraBJeaJYwInOpL7WD5/2wn1oCO7pABQ"}}}; new key: {'key': 'daYxx2uda7KtVrKtxAKanhJ7ip9yzvFEjJzT7Qg0/x0', 'signatures': {'[user_redacted]': {'ed25519:RWDSUMBAID': 'GT9G1g5NTwWPnQGRmf5JcSddW9ZQvHA3Ly3+zs04gnQ4d9OKX5MrbcZ33o27Xd2r43ebZ1a3bDJ/IfUmfH58Cg'}}}
2026-01-11 00:03:01,429 - synapse.access.http.8008 - 521 - INFO - POST-9160297 - [client_ip] - 8008 - {[user_redacted]} Processed request: 0.004sec/0.000sec (0.001sec, 0.000sec) (0.000sec/0.003sec/3) 528B 400 "POST /_matrix/client/v3/keys/upload HTTP/1.0" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.12.7 Chrome/142.0.7444.175 Electron/39.2.3 Safari/537.36" [0 dbevts]

Anything else that would be useful to know?

No response

[localguru]

We have the same problem here. Sometimes a single client hammers our workers with 144 requests per second, which eat up memory until they no longer respond properly.

Regardless of the problem, one could consider rate limits, e.g., directly in nginx. What would be reasonable values for this without affecting valid clients?

[frlan]

I have just seen an issue on a customer's setup with very similar pattern – Upgrade to Element 1.12.9 on client side seems to have fixed it.

[leofah]

Yes, element-web was fast with a hotfix. However this would not solve the underlying problem of the memory leak in synapse. A well crafted request sent 100s of times per second would still DOS the server.

[mcg-matrix]

Regardless of the problem, one could consider rate limits, e.g., directly in nginx. What would be reasonable values for this without affecting valid clients?

I'm running Synapse with nginx as reverse proxy in front of it. Basically every client program (Riot-Android, Element-Android, Element-Web, Element-Desktop, ...) did some accidental busy-looping in some version under whatever conditions.

I've rate-limited requests with the same method + URI + args to:

• 10/minute from the same client address (which may cause trouble if you have many users behind the same NAT address),
• 6/minute within the same connection (client address+port).

My goal was to help prevent the draining of a mobile device's battery. The maximum extra delay of 10 seconds should be bearable for a human waiting for something to move again and shouldn't cause client-side software to give up.

Might be stupid for some reason, might be too aggressive – but it's been running like that for years. I can see (from nginx's log) the rate-limit kicking in every now and then, but I never observed or received a report about something no longer working.