Epic: Gateway state convergence — eliminate UI/backend state drift

[ilblackdragon]

Problem

Every recent UI desync report in the web gateway is the same class: the UI is driven by a stream of deltas without ever reconciling against an authoritative state. Two sources of truth diverge, and nothing forces them back together.

Concrete drift surfaces in play today:

• Engine event log vs. AppEvent broadcast stream — thread_event_to_app_events in src/bridge/router.rs translates 8 of 18 engine variants (tracking: complete engine→AppEvent coverage (gaps after #2571 / #2530) #2654); plan tool, gate manager, and reasoning tool also emit AppEvents without a corresponding engine event, so the engine log and the broadcast stream are not the same projection.
• Broadcast stream vs. UI state — BroadcastStream silently drops on lag (src/channels/web/platform/sse.rs:188-202); subscribe_raw (WebSocket + Responses API) has no Last-Event-ID replay at all.
• UI state vs. GET response — cold-load restoration historically ignored server-reported canonical state ([QA] Web UI: refresh without thread hash restores assistant thread instead of active non-assistant thread #2285, fixed by [codex] Fix web chat refresh active thread #2330 but the shape will recur).
• v1 pending_auth vs. v2 gate_required — two live gate protocols; Web UI: Service connection flow broken (stale approval + lost OAuth callback) #2534's stale approval modal is exactly a modal resolved on one side while chat state stayed on the other.
• Cache vs. truth — WorkspacePool cached by user_id with token-specific scopes applied post-cache (Gateway: pre-existing correctness/perf issues surfaced during #2628 platform extraction #2633 item 1); seed_if_empty runs after the cache insert and a seed failure stays cached forever (Gateway: pre-existing correctness/perf issues surfaced during #2628 platform extraction #2633 item 2).

Fixing each symptom individually is linear work with no upper bound. The structural fix below makes the class finite.

Target invariants

1. One source of truth per concept, everything else is a projection. Turn progression → engine event log. Gates → pending-gate store. Files → workspace. Every other surface (SSE, WS, history API, frontend state) is a pure derivation. No tool or handler may emit an AppEvent without writing an engine event first.

2. Every UI concept has exactly one GET that returns canonical truth. Events are accelerators, not contracts. If the stream is silent for an hour, the UI must converge by calling one documented endpoint. Most GETs already exist (/api/chat/history.pending_gate, .in_progress, /api/chat/threads.active_thread) — the rule is enforcement, not new surface.

3. Client and server share a monotonic cursor per stream. HistoryResponse, ThreadsResponse, and key status GETs return the same boot_id:counter cursor that SSE emits. On reconnect, tab focus, or cursor mismatch, the GET is canonical; the UI refetches the slice.

4. One transport with one envelope. SSE and WS share event-ID semantics and reconciliation protocol, or one is deprecated for chat. The current asymmetry (IDs on SSE, none on WS; two envelope shapes) is free drift.

Program of work (ordered)

Phase 1 — close the bridge (prereq for everything else)

• tracking: complete engine→AppEvent coverage (gaps after #2571 / #2530) #2654 — finish thread_event_to_app_events coverage for the 10 dropped engine variants and the 7 missing semantic variants.
• Add a pre-commit / CI check: broadcast( and broadcast_for_user( outside bridge::thread_event_to_app_events fail unless annotated with a trailing // projection-exempt: <reason>. Pattern mirrors dispatch-exempt in scripts/pre-commit-safety.sh.
• Migrate current non-engine AppEvent emit sites (plan tool, gate manager, reasoning tool, any direct channel broadcasts) to record an engine event first and derive the AppEvent via the bridge.

Phase 2 — cursor + reconcile protocol

• Add cursor: String (same boot_id:counter shape as SSE event IDs) to HistoryResponse, ThreadsResponse, ExtensionsResponse, and the status GETs the frontend polls.
• Frontend reconciler (crates/ironclaw_gateway/static/js/core/...): on (a) SSE reconnect, (b) tab focus, (c) lastEventId vs. GET cursor mismatch — refetch the canonical slice and apply as authoritative, overriding local optimistic state.
• Documentation rule: every AppEvent variant names the GET that returns the same truth. Reviewers enforce.

Phase 3 — replay from the engine log

• GET /api/chat/events/replay?thread={id}&since_cursor={N} — re-derive AppEvents from persisted engine events, not from the in-memory broadcast channel. Reconnect path switches to this.
• Consequence: BroadcastStream silent drop becomes a performance concern (stale buffer), not a correctness bug (lost event).

Phase 4 — unify transports

• Either add event-ID replay semantics to subscribe_raw (WS path) or deprecate WS for the chat surface. Document the choice in src/channels/web/CLAUDE.md and remove the loser in the same PR.

Phase 5 — delete legacy forks

• Retire v1 pending_auth: delete /api/chat/auth-token, /api/chat/auth-cancel, clear_auth_mode* in platform/legacy_auth.rs, and the no-request_id branch in static/js/core/onboarding.js. Gate on engine v1 retirement.
• Gateway: pre-existing correctness/perf issues surfaced during #2628 platform extraction #2633 items 1-2 — WorkspacePool cache key must include a stable hash of applied scopes (or apply scopes outside the cache); seed-before-insert (or in-flight seed marker) closes the divergence window.

Phase 6 — forcing functions

• feat(web): visible Stop control and hard cancel for active chat turns #2121 hard-cancel endpoint. Not for its own sake — because it forces a TurnCancelled engine event to exist, which forces the reconcile path to exist.
• E2E regression covering the protocol: disconnect SSE for longer than the broadcast buffer, reconnect, assert the UI converges to backend state without a manual refresh. Blocks on Phase 3.

Related issues this class subsumes

• tracking: complete engine→AppEvent coverage (gaps after #2571 / #2530) #2654 — engine→AppEvent coverage gaps
• Web UI: Service connection flow broken (stale approval + lost OAuth callback) #2534 — stale approval modal + lost OAuth callback
• [QA] Web UI: refresh without thread hash restores assistant thread instead of active non-assistant thread #2285 — refresh-without-hash restores wrong thread (fixed; the reconcile protocol prevents the next one)
• feat(web): visible Stop control and hard cancel for active chat turns #2121 — no hard cancel for Processing state
• [Bug Bash 4/20] Telegram messages rendered as separate Engine conversations instead of unified thread #2731 — Telegram conversation splitting (upstream of bridge, same single-source-of-truth rule)
• Gateway: pre-existing correctness/perf issues surfaced during #2628 platform extraction #2633 items 1-2 — WorkspacePool cache vs. truth
• fix: web UI messages stuck until refresh — SSE event ordering bug #2079 (historical) — SSE ordering bug, template for the class

Non-goals

• Rewriting SseManager or the AppEvent wire format. Phases 1-3 are additive.
• Replacing the broadcast channel with a durable queue. Replay comes from the engine event log, which is already durable; broadcast stays the fast path.
• Engine v1 work. Phase 5's legacy deletions gate on v1 retirement elsewhere.
• Frontend rendering of newly-bridged events — follow-ups per variant, tracked under tracking: complete engine→AppEvent coverage (gaps after #2571 / #2530) #2654.

Risks

• Invariant Move whatsapp channel source to channels-src/ for consistency #1 has to be enforced by a script. A convention that says "don't call broadcast directly" will be broken within a week without a pre-commit check. Phase 1 must land the check.
• Phase 2 touches every mutation handler. Small per-handler, nontrivial total surface.
• Phase 3 assumes every UI-visible event has a corresponding engine event. That's exactly what Phase 1 guarantees — run in order.
• Tab-focus reconcile could flood the backend at 100+ concurrent users. Cursor compare keeps most reconciles 304-equivalent; measure before Phase 2 frontend lands.
• Scope creep into engine v2. This epic deliberately does not touch engine internals beyond "emit engine events where they should have existed." If a variant requires engine-loop surgery, file a sub-issue and defer.

[ilblackdragon]

Follow-up: .claude/rules/ additions

The epic's four invariants split cleanly: #1 and #2 are ongoing reviewer-enforced rules; #3 and #4 are one-time protocol work. The first pair belongs in .claude/rules/ so new code doesn't re-introduce the drift we're closing.

New file: .claude/rules/gateway-events.md

Covering two locked-in rules:

• No AppEvent emit outside the bridge. Mirrors the dispatch-exempt pattern in tools.md. Lint check: broadcast( / broadcast_for_user( outside bridge::thread_event_to_app_events must carry a trailing // projection-exempt: <reason> or fail pre-commit. State as a target invariant with the current gap (plan tool, gate manager, reasoning tool) called out inline — same convention tool-evidence.md already uses for aspirational rules.
• Every AppEvent variant names the canonical GET that returns the same truth. Reviewer-enforced, same tier as the "Wire-contract field naming" section in types.md. Prevents new events that have no reconciliation path from shipping.

Small addition to .claude/rules/error-handling.md

A one-paragraph entry under Silent-Failure Anti-Patterns naming BroadcastStream::Err(_) => None (and the silent-drop shape generally) alongside .ok()? and unwrap_or_default(). Same pathology: a transport error becomes an empty state the caller can't distinguish from "nothing happened." The rule doesn't ban it — the broadcast channel legitimately drops under lag — it says the compensating reconcile-from-GET path must exist before the drop is allowed to stay silent.

Deliberately out of scope for .claude/rules/

• Cursor / replay protocol details — implementation-specific, belongs in src/channels/web/CLAUDE.md once Phase 2-3 ship.
• Transport unification — one-shot cleanup, not an ongoing invariant.

Sequencing

Land gateway-events.md as part of the Phase 1 PR that wires the lint check, so the rule and the enforcement mechanism arrive together. Avoids a window where the rule is documented but unenforced.

[ilblackdragon]

Refinement: projection rule is multi-source, not engine-only

Pushing back on a too-narrow framing in the body. "Every AppEvent must be projected from an engine event" would force JobStatus, OnboardingState, and Heartbeat through the engine event log — which is wrong. Those have no engine state backing them; the sandbox worker has its own JobEvent log, OAuth/pairing are channel-lifecycle, and Heartbeat is pure SSE transport.

Corrected Invariant #1

Every AppEvent has exactly one producer source, and that source is either (a) a typed event log — EventKind for engine, JobEvent for sandbox worker, a channel-lifecycle log for onboarding/pairing/OAuth — or (b) an explicitly-approved transport-only variant.

Projection surfaces

Source log	Projection fn	Covers
ironclaw_engine::EventKind	bridge/router.rs::thread_event_to_app_events	Turn progression, tool exec, gates, leases, child threads, skills
worker::JobEvent (to formalize)	worker/job.rs — currently inline, should extract	Sandbox job lifecycle, job_* AppEvents
Channel lifecycle logs	features/oauth/, features/pairing/, features/extensions/ — each local	OnboardingState, ExtensionStatus

Transport-only allowlist (no source log)

Heartbeat, StreamChunk. LLM stream chunks are pre-step-completion and genuinely ephemeral; formalizing them into EventKind would pollute the durable log with token-level noise.

Lint rule (Phase 1 PR 2)

broadcast( / broadcast_for_user( outside the three documented projection functions requires // projection-exempt: <reason> with the reason naming either a source log (e.g., projection-exempt: channel-lifecycle, extensions/manager.rs:4951 onboarding_state) or the transport allowlist (projection-exempt: transport-only, heartbeat).

No change to the phased plan. PR 1 still starts with the 5 cheap EventKind drops; the multi-source framing only matters for PR 2 (lint) and PR 3+ (migrations) so they land on the correct rule.

[ilblackdragon]

Phase 1 in flight

• feat(bridge): project 3 dropped engine events to AppEvents #2797 — ✅ merged. First 3 bridge drops (StepFailed, ChildCompleted, CodeExecutionFailed). Established the CodeExecutionFailureCategory wire-enum pattern per types.md.
• feat(safety): projection-exempt lint for gateway event sources #2840 — open. Projection-exempt lint (check feat: Add Google Suite & Telegram WASM tools #9 in scripts/pre-commit-safety.sh) + .claude/rules/gateway-events.md. Diff-based, so existing unannotated emits aren't retroactively broken; rule is enforced on new code.
• feat(bridge): project 7 more engine events to AppEvents #2844 — open. Second bridge slice: 3 lease events + SelfImprovement collapsed variant + OrchestratorRollback. Adds Display for LeaseId on the engine side as a prereq.
• Migrate AppEvent String IDs and state fields to typed newtypes/enums #2799 — follow-up, deferred. Migrate thread_id / parent_thread_id / job_id / mission_id / request_id Strings in AppEvent to typed newtypes; ThreadStateChanged from/to Strings to typed ThreadState wire mirror.

Still on the Phase 1 todo

After #2840 and #2844 land:

• PR 3 — migrate the current direct-emit violations to engine-event-first:
  • src/tools/builtin/plan.rs:137 (plan tool → PlanUpdate)
  • gate manager's direct GateRequired / GateResolved (enables bridging EventKind::ApprovalRequested / ApprovalReceived without duplicate emit)
  • reasoning / credential-injection direct emits
• Baseline-annotate remaining legitimate emit sites (channel-lifecycle, OAuth callback, legacy v1 auth with its delete tag) with // projection-exempt: — can be bundled with PR 3 so the migration and the baseline land together.

Once PR 3 ships, Phase 1 is done and the rule becomes: no new direct sse.broadcast without a named projection-exempt category.