fix(ai/live): add -liveAIHeartbeatHeadersFile for dynamic token rotation without pod restart

[livepeer-tessa]

Problem

The AI gateway's heartbeat to daydream-api uses headers (including the Authorization bearer token) configured once at startup via -liveAIHeartbeatHeaders. When the daydream-api access token is rotated or expires, the heartbeat begins returning 401 Unauthorized errors indefinitely until the pod is restarted with a new token.

This was confirmed in staging (staging-livepeer-ai-gateway-7280) where recurring Invalid access token errors were traced back to the heartbeat call in ai_mediaserver.go:1452:

E0323 17:41:22.192748  ai_mediaserver.go:1452] heartbeat: failed to send heartbeat 401 Unauthorized
heartbeat: response body: {"error":"Authentication failed","code":"AUTH/FAILED","details":{"cause":"Invalid access token"}}

Tracked in: daydreamlive/scope#710

Solution

Add a new -liveAIHeartbeatHeadersFile flag that points to a file containing key:value headers (one per line). The file is re-read on every heartbeat call, enabling automatic token rotation when the file is backed by a Kubernetes secret volume mount (which updates in-place when the secret is rotated).

File headers override static -liveAIHeartbeatHeaders when keys conflict. The existing flag is unchanged.

File format

# Comments supported
Authorization: Bearer <token>
X-Custom-Header: value

Kubernetes usage

# Mount a secret as a file
volumeMounts:
  - name: heartbeat-auth
    mountPath: /run/secrets/heartbeat-headers
    subPath: headers
volumes:
  - name: heartbeat-auth
    secret:
      secretName: ai-gateway-heartbeat-token

# Pass the path to the process
args:
  - -liveAIHeartbeatHeadersFile=/run/secrets/heartbeat-headers

When the K8s secret is rotated via ExternalSecrets (or manually), the mounted file updates automatically and the next heartbeat will use the fresh token — no pod restart needed.

Changes

• core/livepeernode.go: add LiveAIHeartbeatHeadersFile field
• cmd/livepeer/starter/flags.go: add -liveAIHeartbeatHeadersFile flag
• cmd/livepeer/starter/starter.go: wire config field to node
• server/ai_mediaserver.go: add loadHeartbeatHeaders() helper that merges static + file headers (re-read each call); sendHeartbeat uses it

[codecov]

Codecov Report

❌ Patch coverage is 36.36364% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 32.87235%. Comparing base (507a543) to head (6673a4a).

Files with missing lines	Patch %	Lines
server/ai_mediaserver.go	42.10526%	11 Missing ⚠️
cmd/livepeer/starter/starter.go	0.00000%	2 Missing ⚠️
cmd/livepeer/starter/flags.go	0.00000%	1 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##              master       #3887         +/-   ##
===================================================
- Coverage   32.88638%   32.87235%   -0.01403%     
===================================================
  Files            171         171                 
  Lines          42063       42084         +21     
===================================================
+ Hits           13833       13834          +1     
- Misses         27189       27207         +18     
- Partials        1041        1043          +2     

Files with missing lines	Coverage Δ
core/livepeernode.go	75.16779% <ø> (ø)
cmd/livepeer/starter/flags.go	0.00000% <0.00000%> (ø)
cmd/livepeer/starter/starter.go	21.76550% <0.00000%> (-0.02937%)	⬇️
server/ai_mediaserver.go	7.24638% <42.10526%> (+0.60081%)	⬆️

... and 3 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 507a543...6673a4a. Read the comment docs.

Files with missing lines	Coverage Δ
core/livepeernode.go	75.16779% <ø> (ø)
cmd/livepeer/starter/flags.go	0.00000% <0.00000%> (ø)
cmd/livepeer/starter/starter.go	21.76550% <0.00000%> (-0.02937%)	⬇️
server/ai_mediaserver.go	7.24638% <42.10526%> (+0.60081%)	⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.