feat(test): dual-mode live/replay test harness with LLM judge

[ilblackdragon]

Summary

• Add a general-purpose dual-mode test harness (LiveTestHarness) for running E2E tests with live LLM calls (recording traces) or replay from saved traces (deterministic, no API keys)
• TestRigBuilder gains with_config() for real-binary config parity and with_http_interceptor() for live recording support
• First test case: zizmor security scanner against ironclaw's own GitHub Actions workflows
• Engine v2 variant reveals that EffectBridgeAdapter doesn't honor auto_approve_tools config — documents this as a known gap

How it works

# Record a trace (live LLM + real tools):
IRONCLAW_LIVE_TEST=1 cargo test --features libsql --test e2e_live -- --ignored

# Replay from saved trace (deterministic):
cargo test --features libsql --test e2e_live -- --ignored

• Live mode loads real config via Config::from_env() so ENGINE_V2, ALLOW_LOCAL_TOOLS, approval gates all match the real binary
• Session logs (.log) saved alongside trace fixtures (.json) for human inspection and live-vs-replay diffing
• LLM judge verifies response quality semantically in live mode

New files

File	Purpose
tests/support/live_harness.rs	LiveTestHarness, LiveTestHarnessBuilder, judge_response()
tests/e2e_live.rs	zizmor_scan (v1) + zizmor_scan_v2 (engine v2)
tests/fixtures/llm_traces/live/*.json	Recorded traces
tests/fixtures/llm_traces/live/*.log	Session logs

Test plan

• cargo clippy --features libsql --tests — zero warnings
• Existing recorded trace tests pass (e2e_recorded_trace)
• IRONCLAW_LIVE_TEST=1 records trace + session log for v1 and v2
• Replay mode loads trace and passes assertions
• Replay session log can be diffed against live log (identical for deterministic replay)
• V2 test documents approval gate gap (relaxed assertions)

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a dual-mode E2E test harness that supports both live LLM execution with recording and deterministic replay from saved traces. It includes new tests for zizmor security scans and updates the TestRig to better mirror real binary behavior by allowing configuration overrides. Feedback focuses on ensuring UTF-8 safety during string truncation in logs and correctly respecting tool approval configurations to maintain parity with the production environment.

[gemini-code-assist]

The string truncation logic here (&preview[..200]) is not safe for UTF-8 strings. If a multi-byte character happens to cross the 200-byte boundary, this will cause a panic. To fix this, you should use a character-boundary-aware truncation method. The crate already has a utility for this: ironclaw::agent::agentic_loop::truncate_for_preview. Using it will prevent potential panics and ensure the log output is always valid.

                    let short = ironclaw::agent::agentic_loop::truncate_for_preview(&preview, 200);

References

1. When truncating a UTF-8 string, use character-aware methods (e.g., char_indices() or iterating over chars()) to avoid panics caused by slicing in the middle of a multi-byte character.

[ilblackdragon]

Fixed in f7ceb6e — replaced byte-index slicing with char_indices() to find a safe UTF-8 boundary before truncating.

[gemini-code-assist]

Forcing auto_approve_tools to true here overrides the value from the user's configuration loaded via Config::from_env(). This prevents the live test harness from accurately mirroring the real binary's behavior regarding tool approval gates, which contradicts one of the goals of this harness. This makes it difficult to test scenarios involving approvals, such as the one documented in zizmor_scan_v2. This line should be removed to allow the configuration from the environment to be respected. A corresponding change is needed in tests/support/test_rig.rs to change the default for auto_approve_tools to None.

[ilblackdragon]

Fixed in f7ceb6e — removed forced .with_auto_approve_tools(true) from build_live(). The harness now respects Config::from_env() by default. Added .with_auto_approve_tools() to the builder so tests can opt in explicitly.

[gemini-code-assist]

The default for auto_approve_tools should be None instead of Some(true). When with_config() is used in the live test harness, this will ensure that the value from the loaded configuration is used by default, rather than being overridden. This change is necessary for the live test harness to correctly reflect the real binary's approval gate behavior.

Suggested change

	auto_approve_tools: Some(true),
	auto_approve_tools: None,

[ilblackdragon]

The default Some(true) is kept for backward compat with all existing tests. When with_config() is used, the live harness now controls auto_approve via its own builder method, which applies the override to the config before TestRig sees it.

[Copilot]

Pull request overview

This PR introduces a dual-mode E2E test harness that can either (1) run with live LLM/tool calls while recording traces, or (2) deterministically replay from committed trace fixtures (no API keys), and adds initial zizmor-based live/replay E2E scenarios.

Changes:

• Extend TestRigBuilder to support real-binary config parity (with_config) and direct HTTP interceptor injection (with_http_interceptor).
• Add LiveTestHarness to orchestrate live recording vs replay runs, plus an optional LLM-based judge in live mode.
• Add an ignored e2e_live integration test and commit initial live/replay trace + log fixtures.

Reviewed changes

Copilot reviewed 6 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/support/test_rig.rs	Adds config override + HTTP interceptor override paths to align test rigs with real binary behavior and enable recording/replay.
tests/support/mod.rs	Exposes the new live_harness module behind libsql.
tests/support/live_harness.rs	Implements the dual-mode harness and a simple “judge” helper.
tests/e2e_live.rs	Adds ignored live/replay E2E tests for a zizmor scan (v1 + v2 variant).
tests/fixtures/llm_traces/live/zizmor_scan.json	Recorded trace fixture for replay.
tests/fixtures/llm_traces/live/zizmor_scan.log	Session log fixture (currently labeled Replay).
tests/fixtures/llm_traces/live/zizmor_scan_v2.json	Recorded trace fixture for engine v2 scenario (includes HTTP capture).
tests/fixtures/llm_traces/live/zizmor_scan_v2.log	Session log for engine v2 scenario.
tests/fixtures/llm_traces/live/.gitkeep	Keeps the live fixture directory present in git.

Comments suppressed due to low confidence (1)

tests/support/test_rig.rs:598

• build(self) destructures self into locals, but later in this function it still references self.keep_bootstrap (see around line ~897). After destructuring, self is moved, so this will not compile. Use the destructured keep_bootstrap local consistently (or avoid destructuring / keep self intact).

        // Destructure self up front to avoid partial-move issues.
        let TestRigBuilder {
            trace,
            llm,
            config_override,
            max_tool_iterations,
            injection_check,
            auto_approve_tools,
            enable_skills,
            enable_routines,
            http_exchanges: explicit_http_exchanges,
            http_interceptor_override,
            extra_tools,
            wasm_tools,
            keep_bootstrap,
            engine_v2,
        } = self;

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

When config_override is present, this branch ignores the builder’s with_engine_v2() flag (engine_v2 local) and relies solely on the provided config. This makes with_engine_v2() a no-op in config-parity mode, which is surprising for callers; consider applying engine_v2 when explicitly set, or document that with_config() takes precedence for engine selection.

[ilblackdragon]

Fixed in f7ceb6e — the config-override branch now applies the builder's engine_v2 flag when explicitly set.

[Copilot]

build_live claims to mirror real-binary behavior (including approval gates), but it unconditionally forces .with_auto_approve_tools(true), overriding whatever Config::from_env() resolved. To keep parity, consider defaulting to the config value (don’t call with_auto_approve_tools here) and add an explicit builder knob to override when tests want to bypass approvals.

Suggested change

	// - auto_approve_tools controls approval gates
	let rig = TestRigBuilder::new()
	.with_config(config)
	.with_llm(llm)
	.with_http_interceptor(http_interceptor)
	.with_max_tool_iterations(self.max_tool_iterations)
	.with_auto_approve_tools(true)
	// - auto_approve_tools remains driven by the resolved config
	let rig = TestRigBuilder::new()
	.with_config(config)
	.with_llm(llm)
	.with_http_interceptor(http_interceptor)
	.with_max_tool_iterations(self.max_tool_iterations)

[ilblackdragon]

Fixed in f7ceb6e — removed forced .with_auto_approve_tools(true) from build_live(). The harness now respects Config::from_env() by default. Added .with_auto_approve_tools() to the builder so tests can opt in explicitly.

[Copilot]

LiveTestHarnessBuilder stores timeout, but it’s never read/used when building the rig or waiting for responses. This can mislead callers into thinking it changes harness behavior; either wire it into the harness API (e.g., expose as default for wait_for_responses / internal timeouts) or remove it.

[ilblackdragon]

Fixed in f7ceb6e — removed the timeout field and with_timeout() method.

[Copilot]

judge_response treats any output starting with PASS as a pass, even if the judge doesn’t follow the required PASS: / FAIL: one-line format. This makes false positives more likely (e.g., PASSING ... or multi-line output). Consider parsing strictly for PASS: / FAIL: (case-insensitive) and treating anything else as a failure with a helpful reasoning message.

Suggested change

	let pass = trimmed.starts_with("PASS");
	JudgeVerdict {
	pass,
	reasoning: trimmed.to_string(),
	}
	if trimmed.is_empty() {
	return JudgeVerdict {
	pass: false,
	reasoning: "Judge returned empty output; expected exactly one line starting with PASS: or FAIL:".to_string(),
	};
	}
	if trimmed.lines().count() != 1 {
	return JudgeVerdict {
	pass: false,
	reasoning: format!(
	"Judge returned invalid multi-line output; expected exactly one line starting with PASS: or FAIL:. Raw output: {}",
	trimmed
	),
	};
	}
	let line = trimmed;
	if let Some(reasoning) = line
	.get(..5)
	.filter(|prefix| prefix.eq_ignore_ascii_case("PASS:"))
	.map(|_| line[5..].trim())
	{
	JudgeVerdict {
	pass: true,
	reasoning: reasoning.to_string(),
	}
	} else if let Some(reasoning) = line
	.get(..5)
	.filter(|prefix| prefix.eq_ignore_ascii_case("FAIL:"))
	.map(|_| line[5..].trim())
	{
	JudgeVerdict {
	pass: false,
	reasoning: reasoning.to_string(),
	}
	} else {
	JudgeVerdict {
	pass: false,
	reasoning: format!(
	"Judge returned invalid output; expected exactly one line starting with PASS: or FAIL:. Raw output: {}",
	trimmed
	),
	}
	}

[ilblackdragon]

Fixed in f7ceb6e — now parses strictly for PASS: / FAIL: prefix via strip_prefix(). Any other format is treated as failure with a diagnostic message.

[Copilot]

This trace fixture embeds absolute local paths (e.g. /Users/coder/ironclaw8 and /Users/coder/.cargo/bin/zizmor) inside recorded tool outputs. That leaks machine-specific info and makes fixtures harder to share/re-record across environments. Consider scrubbing/normalizing paths in recorded fixtures (or in the recorder) before committing (e.g., replace with relative paths or a placeholder root).

[ilblackdragon]

Acknowledged — these are inside recorded tool output (what the LLM saw). Scrubbing would make replay diverge from reality. Will consider path normalization / HTTP body trimming in the recorder as a follow-up.

[Copilot]

The recorded http tool result stores a large HTML body plus many volatile headers (date, x-github-request-id, etc.). This tends to bloat the repo and causes frequent fixture churn on re-recording without improving test coverage. Consider redacting/truncating HTTP bodies and dropping non-essential headers in the recording layer (or post-process fixtures) so replay remains deterministic but compact and stable.

[ilblackdragon]

Acknowledged — these are inside recorded tool output (what the LLM saw). Scrubbing would make replay diverge from reality. Will consider path normalization / HTTP body trimming in the recorder as a follow-up.