[daydream-api] Recurring 'Invalid access token' errors in staging — ~6x/12h with hourly-ish pattern

[livepeer-tessa]

Summary

The daydream-api (staging, eu-metrics-monitoring.livepeer.monster) is logging 6 recurring [AuthPlugin] Invalid access token errors over a 12-hour window on 2026-03-17. The regularity suggests an automated client making periodic requests with an expired or invalid token.

Error Details

Loki query used: {app="daydream-api"} | logfmt | level="error"
Environment: Staging
Hostname: d896435a6ee678
Log level: pino level=50 (ERROR)

Occurrences in the 12h window (06:09–18:09 UTC, 2026-03-17):

[2026-03-17 07:09:36 UTC] [AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token
[2026-03-17 08:54:09 UTC] [AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token
[2026-03-17 12:39:02 UTC] [AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token
[2026-03-17 14:30:12 UTC] [AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token
[2026-03-17 16:00:33 UTC] [AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token
[2026-03-17 17:52:43 UTC] [AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token

Intervals between occurrences: 1h45m → 3h45m → 1h51m → 1h30m → 1h52m — somewhat regular, consistent with an automated process.

Sample Log Line

level=50 time=1773769963362 pid=670 trace_id=ec52a5f5425e56c7e0c9f29de4e2f747 span_id=e7888e9e5392feae trace_flags=01 msg="[AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token"

Likely Cause

An automated client (monitoring script, health checker, or Scope app with a stale session token) is making periodic requests to staging daydream-api using a token that is expired or has been revoked. The server correctly rejects it, but this pollutes the error log and may indicate:

1. A service account token needs rotation
2. A client isn't properly refreshing/renewing tokens
3. A test/monitoring script has a hardcoded expired token

Impact

• Inflates error count in staging dashboards (6 false alarms / 12h)
• Could mask real auth failures from legitimate users

Next Steps

1. Identify which client/service account is making these requests (trace the trace_id in the full daydream-api logs — each occurrence has a unique trace ID)
2. If it's a monitoring script: rotate the token
3. If it's a Scope client: investigate token refresh logic in the AuthPlugin

Automatically detected by Scope error monitor cron — 2026-03-17 18:09 UTC

[livepeer-tessa]

2026-03-23 Update — AI Gateway Heartbeat Identified as Auth Failure Source

Today (2026-03-23 17:41 UTC) another Invalid access token error was recorded, and this time the trace crosses two services:

ai_mediaserver.go (staging-livepeer-ai-gateway-7280):

E0323 17:41:22.192748  ai_mediaserver.go:1452] heartbeat: failed to send heartbeat 401 Unauthorized
E0323 17:41:22.192825  ai_mediaserver.go:1453] heartbeat: response body: {"success":false,"error":"Authentication failed","code":"AUTH/FAILED","status":401,"details":{"cause":"Invalid access token"},"requestId":"req-95q8","traceId":"712232328e38c2750ecdebe32d8f735a"}

daydream-api (same traceId):

level=50 time=1774287682159 pid=670 trace_id=712232328e38c2750ecdebe32d8f735a msg="[AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token"

Confirmed source: The AI gateway's periodic heartbeat to daydream-api is using an invalid/expired access token. The token the AI gateway is configured with for its heartbeat endpoint needs to be rotated or the token refresh mechanism needs to be checked in ai_mediaserver.go.

Detected by Scope error monitor cron — 2026-03-23 18:09 UTC

[livepeer-tessa]

Fix Submitted — livepeer/go-livepeer#3887

Root cause confirmed: the -liveAIHeartbeatHeaders flag writes headers into a static map at pod startup, so a rotated/expired daydream-api token cannot be picked up without a pod restart.

PR: livepeer/go-livepeer#3887

What the fix does: Adds a -liveAIHeartbeatHeadersFile flag. The file is re-read on every heartbeat call, so mounting a K8s secret as a volume file allows token rotation without pod restarts. Backward-compatible — existing -liveAIHeartbeatHeaders flag is unchanged.

Immediate mitigation (staging): Rotate the token in Vault → ExternalSecret will push the updated K8s secret → restart the staging-livepeer-ai-gateway-7280 pod to pick up the new value (one-time restart needed until the PR ships and the Helm chart is updated to use liveAIHeartbeatHeadersFile).

Updated by Scope error monitor cron — 2026-03-23 18:15 UTC

[livepeer-tessa]

New Occurrences — 2026-03-26

4 more [AuthPlugin] Failed to verify auth token errors observed in today's 12h window (06:09–18:09 UTC):

Time (UTC)	Hostname	trace_id
06:31:34	d896435a6ee678	15044bde73d6284f831e59cb6e70933d
07:54:49	d896435a6ee678	71c84f96a65b9ab0deb0681296d9b9dc
08:01:31	d896435a6ee678	cd4ea6826829819d030c354c0676d2da
13:02:06	2876702b0e6558	8b11191a218826d9f26fe8b14d28c93d

Intervals roughly match prior pattern. Still staging only. Pattern continues unabated since 2026-03-17 — appears to be an automated client with a token that's not being refreshed.

Detected by automated Scope/fal.ai error monitor — 2026-03-26 18:09 UTC

[livepeer-tessa]

2026-03-28 Update (6:09 AM UTC)

Pattern persists. 4 new occurrences in the last 12-hour window (18:09 UTC Mar 27 → 06:09 UTC Mar 28):

[2026-03-27 18:40:51 UTC] [AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token
[2026-03-27 21:06:30 UTC] [AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token
[2026-03-28 00:06:33 UTC] [AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token
[2026-03-28 04:01:29 UTC] [AuthPlugin] Failed to verify auth token: HTTPError: Invalid access token

Spacing: ~2h26m → ~3h → ~3h55m — still consistent with an automated client rotating but not refreshing its token. Hostnames: 2876702b0e6558 and d896435a6ee678 (both staging daydream-api instances).

This has now been observed across multiple days with similar frequency. Worth identifying which integration/client is sending these requests.