Trajectory benchmarks and e2e trace test rig

[zmanian]

Summary

• Add e2e trace test rig (TraceLlm + TestRig) for replaying LLM traces against the full agent loop
• Unified trace format around turns -- each turn pairs a user_input with LLM response steps, making traces self-contained conversation trajectories
• Backward-compatible: flat "steps" JSON (legacy) is deserialized as a single turn transparently
• Add TestRig::run_trace() for automatic multi-turn replay (send user messages, collect responses, advance turns)
• Add trace fixtures across three tiers: spot/ (quick smoke tests), coverage/ (tool/feature coverage), advanced/ (multi-turn, error recovery, steering, iteration limits, prompt injection)
• Add advanced e2e tests: multi-turn memory coherence, user steering, tool error recovery, long tool chains, workspace search, iteration limit guard, prompt injection resilience
• Add benchmark design plans and phased implementation docs
• Remove in-tree benchmark harness, retain retain_only utilities
• Add GitHub Actions benchmark workflow (manual trigger)
• Add --json flag for machine-readable benchmark output
• Add SkillRegistry::retain_only and wire skill filtering in scenarios

Test plan

• cargo test trace_llm -- unit tests for trace types, deserialization (flat + turns), backward compat
• cargo test e2e_spot -- spot check smoke tests pass
• cargo test e2e_advanced -- multi-turn, steering, error recovery, long chains, memory, injection tests pass
• cargo test e2e_status -- status event tests pass
• Trace fixtures cover: greeting, math, tool echo, JSON ops, chain write/read, memory save/recall, error paths, shell, list_dir, apply_patch, injection detection, multi-turn steering, workspace search, iteration limits

Generated with Claude Code

[ilblackdragon]

Checked that "check_telegram" test actually works and hits it with coverage:

[zmanian]

Review of Illia's commits (a53a852..49dfad6)

Overall these are solid improvements -- declarative expects, AppBuilder integration, and test deduplication are all good moves. Three issues worth discussing:

1. Duplicate type systems between src/llm/recording.rs and tests/support/trace_llm.rs

Both files define parallel trace types (TraceStep, TraceResponse, TraceToolCall, RequestHint, etc.) that are structurally similar but not shared. Changes to the trace format need to be kept in sync in two places. We should either:

• Move the canonical types into src/ (behind a testing feature) so both recording and replay use the same types
• Have trace_llm.rs import from recording.rs

2. Recording produces flat steps, not turns

RecordingLlm emits UserInput markers in a flat steps array. The LlmTrace deserializer wraps this into a single turn with placeholder user input "(test input)". For multi-turn recorded traces, this means run_trace() sends only the placeholder message, not the actual user messages from the recording. The UserInput markers are in the step list but get skipped by TraceLlm.

This works today because flat steps replay correctly in sequence, but run_and_verify_trace() won't actually drive multi-turn conversations from recorded traces -- it'll send one placeholder message and replay all steps linearly.

Should RecordingLlm produce turns instead? Or should the deserializer split flat steps at UserInput boundaries into separate turns?

3. with_tools() / with_workspace() removal

The switch to AppBuilder::build_all() is great for prod/test parity, but removing with_tools() means tests can no longer control tool registry composition. All tests now get the full tool set. This is fine for E2E realism, but tests like iteration_limit_stops_runaway that previously used a minimal registry may be slower or have unexpected interactions. Worth keeping an eye on.