Symptom
Automatic rotation was not reliable for 24/7 recovery.
The watcher detected the rate-limit and started rotation, but the machine could remain on the blocked account:
[2026-03-29 08:04:55] [watcher] Rate-limit detected -> full rotation
[2026-03-29 08:04:57] [OCI] Rotation accepted (attempt=1)
[2026-03-29 08:10:22] [OCI] /result poll timeout - rotation may still be running
[2026-03-29 08:10:23] [rotate_callback] OCI result poll timed out - rotation may have failed
[2026-03-29 08:10:33] [watcher] All 1 account(s) blocked via rateLimitResetTimes
A later second trigger was needed before rotation finally recovered.
Root Cause
watcher_runner.py only ran the local Mac fallback under:
if not success and not oci_up:
success = _run_local_fallback()So if OCI was reachable and accepted the job, but later failed to deliver fresh credentials (/result timeout, browser OAuth failure, inject failure), the same callback never executed a local fallback.
There was a second bug in the same path: if a later local fallback succeeded after an OCI attempt, the success block still treated the source as oci, which misrouted source/telemetry/fleet-push handling.
Fix
- fallback condition changed to
if not success and (not oci_up or used_oci)
- added
used_local_fallback so the real success source is tracked correctly
- success log, telemetry, fleet-push, and opencode session resume now use
mac_fallback when local fallback actually saved the run
- raised AppleScript timeout in
core/opencode_restart.py from 10s to 25s for the session-resume path
Files:
antigravity/watcher_runner.pyantigravity/core/opencode_restart.py
Verification
Targeted smoke test for OCI accepted -> poll timeout -> local fallback success now produces:
WARN: [rotate_callback] OCI result poll timed out - rotation may have failed
WARN: [rotate_callback] OCI path did not yield fresh credentials - running Mac fallback.
INFO: [rotate_callback] SUCCESS (Mac fallback). Local rotations this hour: 1
INFO: telemetry:mac_fallback:
INFO: fleet:mac_fallback:local@example.com
INFO: resume:mac_fallback
This proves the same callback now recovers automatically instead of waiting for a later trigger.
Symptom
Automatic rotation was not reliable for 24/7 recovery.
The watcher detected the rate-limit and started rotation, but the machine could remain on the blocked account:
A later second trigger was needed before rotation finally recovered.
Root Cause
watcher_runner.pyonly ran the local Mac fallback under:So if OCI was reachable and accepted the job, but later failed to deliver fresh credentials (
/resulttimeout, browser OAuth failure, inject failure), the same callback never executed a local fallback.There was a second bug in the same path: if a later local fallback succeeded after an OCI attempt, the success block still treated the source as
oci, which misrouted source/telemetry/fleet-push handling.Fix
if not success and (not oci_up or used_oci)used_local_fallbackso the real success source is tracked correctlymac_fallbackwhen local fallback actually saved the runcore/opencode_restart.pyfrom 10s to 25s for the session-resume pathFiles:
antigravity/watcher_runner.pyantigravity/core/opencode_restart.pyVerification
Targeted smoke test for
OCI accepted -> poll timeout -> local fallback successnow produces:This proves the same callback now recovers automatically instead of waiting for a later trigger.