feat: AI-powered tool improvement suggestions via Claude Code Action

[LOCKhart07]

Summary

• Adds a suggest-improvements job to the daily benchmark workflow that uses Claude Code Action to identify the worst-performing prediction tool, diagnose prompt issues, apply fixes, and open a PR
• Human-in-the-loop: PRs opened by Claude are suggestions, not final — a reviewer must run cached replay to validate improvements before merging
• New benchmark/prepare_suggestion.py pre-extracts target tool context (stats, calibration, overconfidence, platform breakdown, comparison tools) into a single file, saving Claude ~5 turns of discovery
• benchmark/scorer.py enriched with parse_breakdown_by_tool and overconfidence_by_tool for richer diagnostics
• benchmark/CLAUDE.md provides tool registry, scoring reference, data-backed improvement patterns (base-rate anchoring, tail discipline, overconfidence fixes), and guidelines for the AI agent
• Graceful turn limit handling: if Claude completes all phases but hits the turn limit, the job still succeeds
• Dedup check prevents duplicate PRs/issues for the same tool
• Falls back to opening a GitHub issue when no concrete fix is found
• suggest-improvements uses same-run scores.json (via actions/download-artifact@v4) so it always targets the correct worst tool
• Prompt enforces branch creation (auto-improve/<tool-name>) — never commits directly to main
• Prompt specifies exact Phases Completed: All log format so the outcome check works reliably
• Slack notification step has continue-on-error: true so failures don't block artifact uploads

Design Decisions

• Claude Code Action over a lighter script: Prompt improvement requires reading multiple tool source files, comparing patterns across tools, and making judgment calls about what to change — tasks that benefit from an LLM agent with code access. A rule-based script can't diagnose "weak reasoning scaffolding" or "missing bias correction."
• Human-in-the-loop over auto-merge: MVP approach to save CC turns. Claude opens a PR with its diagnosis and fix; a human reviewer validates via cached replay before merging. This avoids the cost of running replay in CI on every suggestion, while keeping a safety gate on prompt changes.
• Pre-extracted context (prepare_suggestion.py): Without this, Claude spends ~5 turns discovering files, reading scores, and figuring out which tool to target. Pre-extracting saves turns (and API cost) by handing Claude everything it needs upfront.
• One tool per run: Each daily run targets only the single worst-performing tool. This keeps runs focused, avoids conflicting changes, and makes PRs easy to review.
• Scheduled runs have suggestions on by default, manual dispatch defaults to off. The repo variable ENABLE_TOOL_SUGGESTIONS acts as a kill switch if needed.

Controls

• run_suggestions workflow input toggle (defaults to off for manual dispatch)
• ENABLE_TOOL_SUGGESTIONS repo variable kill switch for scheduled runs
• BENCHMARK_ANTHROPIC_API_KEY secret required

Tested on fork

• Verified end-to-end on LOCKhart07/mech-predict
• Latest run (7-day prod data): workflow run — all jobs passed, Claude correctly identified prediction-request-rag-claude as worst tool and opened PR #3
• Earlier run: workflow run — Claude targeted superforcaster and opened PR #2

Test plan

• Trigger manually with run_suggestions: true
• Verify suggestion_log.txt artifact is uploaded
• Check that Claude opens PR or issue with auto-improvement label
• Reviewer runs cached replay on the PR to validate before merging
• Verify dedup: re-run should skip if PR/issue already open
• Test kill switch: set ENABLE_TOOL_SUGGESTIONS=false

🤖 Generated with Claude Code