Presence ghosts in entryway etc. as of 9/5 smoke test for 2025

[apjanke]

For 2025, as of 9/5 and the smoke test, there are presence ghosts of several users in the entryway, theater (The Big Top), and obelisk rooms. These are users from previous years' events that are still showing up there.

Theater/Big Top:

Obelisk:

These showed up after we changed the roomIds for ticketBooth and theBigTop to entryway and theater, respectively. Those roomIds are special, and need to be present, along with obelisk. I think the rest of the rooms for this year all have roomIds that weren't used last year. I remember the two ghosts in Obelisk being there all along; I'm guessing because that was a reused roomId all along.

References

• Presence counts are slightly wonky #541 "Presence counts are slightly wonky" from Oct 2022
  • Describes a case where there were excess idle users still in some rooms, and it was fixed by manual presence data deletion.
• Detect if users are AFK and note them as such #211 "Detect if users are AFK and note them as such"

[apjanke]

I think:

The state that indicates which users are in what room is in the Redis, in the presenceData, theaterPresence, and other tables. That probably needs to be cleared out.

Rebooting the whole Redis would fix that. But also probably break other stuff, maybe losing things like mod and speaker status, name badge contents, etc. So we need to be selective. From Discord:

I think the users being in a room vs. not being anywhere is the user's "active" state. That's controlled by the setUserAsActive AFA endpoint, which is done with a false in heartbeat, banUser, and disconnect.

$ grepx setUserAsActive
./server/src/endpoints/heartbeat.ts:43:    await DB.setUserAsActive(user, false)
./server/src/endpoints/banUser.ts:60:  await DB.setUserAsActive(target, false)
./server/src/endpoints/disconnect.ts:7:  await DB.setUserAsActive(user, false)
./server/src/heartbeat.ts:29:  await DB.setUserAsActive(user, true)
./server/src/redis.ts:93:  async setUserAsActive (user: User, isActive: boolean = true) {
./server/src/cosmosdb.ts:67:  async setUserAsActive (user: User, isActive: boolean) {
./server/src/database.ts:34:  setUserAsActive(user: User, active: boolean);

So I'm guessing the users we see here are those whose browsers closed abruptly or something else, such that the `disconnect didn't get called for them, and they stay active.

I don't see an admin interface for "make user Inactive" like there is for "ban user". Over in #541, Em describes a similar situation that was fixed by deleting the list of people for the theater, using del theaterPresence. I think that is a Redis command that is issued on the Redis instance in Azure via the Azure Portal, not the MUD interface.

It looks like the heartbeat endpoint has server-side logic to make idle users inactive, and it operates on more users than just the user who called the heartbeat endpoint. It's based on the per-user timestamp for each users in getHeartbeatData(). Don't know why it didn't cause these users to be cleaned up. ... Hmm: maybe because it's based on the users with values in heartbeatData, if a user wasn't connected very long or there was a problem, they won't have an entry in heartbeatData, and won't get detected by this cleanup logic? Or maybe for some of them, getFullUser(userId) is returning null, so they get skipped.

From server/src/endpoints/heartbeat.ts:

  const data = await getHeartbeatData()
  const now = new Date()
  const nowValue = now.valueOf()

  const activeUsers = []
  const usersToRemove = []
  Object.keys(data).forEach((user) => {
    const time = data[user]
    const diff = nowValue - time
    if (Math.floor(diff / 1000) > thresholdSeconds) {
      usersToRemove.push(user)
    } else {
      activeUsers.push(user)
    }
  })
[...]

  for (let i = 0; i < usersToRemove.length; i++) {
    const userId: string = usersToRemove[i]
    const user = await getFullUser(userId)

    if (!user) continue

    await DB.setUserAsActive(user, false)
    log('Set user as inactive')
    groupManagementTasks.push({userId, groupId: user.roomId, action: 'remove'})
  }

Possible fixes

I think we may need to do something similar to #541, and set all the old users as inactive, either by directly selectively manipulating the Redis data, or maybe adding some new admin tool. E.g. there's a moveAllUsersToEntryway endpoint that changes everyone's location, but AFAICT no similar makeAllUsersInactive. Could create one.

Based on what I've read, I think that to clear room presence data, you can delete the whole presence table for a room (like in Em's #541 there), and Redis will auto-vivify it as an empty table next time it's used. Those users would still be in the active users list, but not present in rooms. For these ghosts specifically, that would be, in Redis:

del theaterPresence
del entrywayPresence
del obeliskPresence

I don't have permissions to do that on our Azure, but maybe one of the other organizers could do that?

Maybe the admin Close the Space operation should also make all users inactive and remove all users from all rooms? As a way of getting a clean slate and clearing up presence ghosts like this. That might be fun to have everyone appear in the entryway when the space is opened, too; that's how it works when a real circus opens.

For a more robust fix, see why the endpoints/heartbeat.ts endpoint cleanup logic isn't picking up these users. Maybe, like, the heartbeatData should get an initial value from connect instead of waiting for a heartbeat execution, so users are always present there. Or it should get its list of users from a bigger list of users instead of just heartbeat.

[apjanke]

Looks like connect is doing an intial user heartbeat early on, so there's a pretty short time between a user first connecting and having a heartbeat recorded. i.e., we're already doing that "populate initial heartbeat upon connection" I was talking about.

If these ghosts are indeed no-heartbeat-data users, then that probably came from an exception case or network problem instead of e.g. the browser going away shortly after connection. One possible error case: if there's no entryway room defined, the user falls back to the default, and the DB.setCurrentRoomForUser() call fails and raises an exception instead of returning an error result object?

From server/src/endpoints/connect.ts:

const connect: AuthenticatedEndpointFunction = async (user: User, inputs: any, log: LogFn) => {
  log('We have a user!', user.id)
  const result: Result = {}

  let room = await DB.getRoomData(user.roomId)

  // WARNING: For now, checking the existence of a roomId in roomData is a good safeguard
  // But this may bite us when/if we ever have programmatic room creation
  if (user.roomId === undefined || !room) {
    user.roomId = 'entryway'
    room = await DB.getRoomData(user.roomId)
  }

  await DB.setCurrentRoomForUser(user, user.roomId)
  await userHeartbeatReceived(user)

[apjanke]

Looking at the prod Redis, the counts of values in the keys for theaterPresence, obeliskPresence, and entrywayPresence look like they match the number of users look like they match the users displayed in the MUD website.

I don't know the Redis data structure enough to find the heartbeat and "user is active" data and correlate them.

Seems consistent with "do del entrywayPresence" as an ops workaround for the current presence ghosts. But I don't know if it's wise to do that so close to the Preview event.

[apjanke]

Close look at specific example

I connected to Redis and took a close look at one of the users that's a presence ghost. Gregor is present in entryway. His userId is in entrywayPresence, but not in activeUsersList. So he's inactive but in a room.

• User: Gregor
• id: SSbOZwjnXze5j019wocCupGyr4C3

Profile, gleaned from browser dev tools:

{"user":{"username":"Gregor","realName":"","pronouns":"","description":"","askMeAbout":"","twitterHandle":"","url":"","hostname":"https://roguelikecelebration2021.azurewebsites.net","id":"SSbOZwjnXze5j019wocCupGyr4C3","roomId":"entryway","unlockedBadges":[{"emoji":"🐣","description":"Attended the 2022 preview event!"},{"emoji":"9️⃣","description":"Attended the 9th Roguelike Celebration in 2024!","category":"YEAR_2024"}]}}

Redis info:

> GET SSbOZwjnXze5j019wocCupGyr4C3Heartbeat
"1662943080279"

>smembers activeUsersList
1) "7iTzPXfVFUSlCp3nIotA4hNRBDi1"
2) "wmSxNEOF6pZ35BFJvkD1kLlS3g83"

>smembers entrywayPresence
 1) "mU6bApCbaDObTfEaxam2SVJbCkI3"
 2) "SSbOZwjnXze5j019wocCupGyr4C3"
[...]

That heartbeat value, as a Unix timestamp, is 2022-09-12. His userId is in entrywayPresence, but not in activeUsersList. (The two users in activeUsersList there are two of my users, intoverflow and alice.)

Causes

So I don't know the specifics of it, but maybe a race condition or a bug or network failure caused him to get removed from activeUsers but not the roomPresence list. Or get re-added to room presence, e.g. if a room move happened about the same time as a disconnect. Maybe some of that in combination with a multipe-browser-tabs situation.

Either way, once you're in this state, the heartbeat cleanup logic is no longer going to clean you up, because the getHeartbeatData() it uses to get candidate users to clean uses the activeUsersList set to get the list of candidate users. He's not in that, so will never be a cleanup candidate there. So I think his ghost is stuck in this room until he actually reconnects and gets added back to activeUsers, and has a live connection that will do disconnect logic, etc.

The big difference here from my speculation in Discord earlier is that heartbeat's starting list for users to maybe clean up isn't "users with heartbeat data", it's "users in activeUsersList". So, having it get the user list from a different source could solve that. But could be a performance problem: I think that heartbeat cleanup logic runs every time any user does a heartbeat, so starting with allUserIds might be a performance problem.

I don't know what the code fix is to prevent this from happening in the first place. But this seems consistent with what we've seen and been thinking so far. And seems like the workaround of clearing the roomPresence Redis keys (like with del entrywayPresence) would work, and probably be pretty safe?