Ranking server sometimes loses some subchanges

[kunyavskiy]

Description:

During IOI 2024 contest, we had a live stream, using scoreboard as a source of data for inforgrafics.
In particular, we are using /subchanges endpoint to receiver score updates. While not being optimal, for simplicity,
we just reload the whole endpoint every 5 seconds.

The problem is on the second day by some reason some of day1 submissions sumtimes disappeared from /subchanges endpoint and appeared back.

As an estimation here is log of 4-digit submission ids available requested several times.

/tmp/d1/message$ curl https://ranking.ioi2024.eg/subchanges/ | grep -oe '"submission": "...."' | wc -l
5383
/tmp/d1/message$ curl https://ranking.ioi2024.eg/subchanges/ | grep -oe '"submission": "...."' | wc -l
5383
/tmp/d1/message$ curl https://ranking.ioi2024.eg/subchanges/ | grep -oe '"submission": "...."' | wc -l
5373
/tmp/d1/message$ curl https://ranking.ioi2024.eg/subchanges/ | grep -oe '"submission": "...."' | wc -l
5373
/tmp/d1/message$ curl https://ranking.ioi2024.eg/subchanges/ | grep -oe '"submission": "...."' | wc -l
5383

There is normally 5383 submissions with 4-digit id, while 10 of them disappeared for some time, and then appeared back.

I'm not sure if it is cms bug, or setup problem, as I don't have any access to setup.

Expected: All submissions of the first day are always available during the second

Actual: They eventually disappears and appears back

Logs

System Information

Unfortunately, I don't have access to setup, so I have no logs.

[prandla]

I reviewed the RWS code and I can't find any reason why it'd sometimes not send existing events. The only explanation that seems likely to me is if there were multiple instances of RWS behind a load balancer, and one of them didn't get some of day 1's submissions (which could be caused by network instability or something). do we have anyone from last year's TC who can confirm or deny this hypothesis?