feat(evals): add reliability harvester and 500/503 retry support

[alisa-alisa]

Summary

This PR implements a robust retry mechanism for transient API errors (500 and 503) encountered during evaluation runs. It also introduces an automated harvester script to gather and analyze these reliability events from GitHub Actions artifacts, helping us track and reduce developer friction caused by API instability.

Details

• Retry Logic: Updated evals/test-helper.ts to automatically retry tests up to 3 times when the Gemini API returns INTERNAL (500) or UNAVAILABLE (503) errors.
• Reliability Logging: Every retry or skip event is logged to evals/logs/api-reliability.jsonl with timestamps, test names, model info, and error codes.
• Harvester Script: Added scripts/harvest_api_reliability.sh which uses the GitHub CLI (gh) to download artifacts from recent E2E and Nightly eval runs and generate a summary report of API reliability.
• Security Enhancements: Refactored test file setup to explicitly check for and block absolute paths or directory traversal attempts in test case definitions.
• CI Configuration:
  • Disabled Vitest's internal retry mechanism for evals (VITEST_RETRY: 0) to ensure our custom logging-aware retry logic takes precedence.
  • Added artifact upload for reliability logs in both chained E2E and nightly eval workflows.

Related Issues

Resolves #23168.

How to Validate

1. Unit Tests: Run npx vitest evals/test-helper.test.ts to verify the retry logic, logging, and security checks.
2. Reliability Log: Run a test that fails with a 500 error (e.g., by mocking) and verify evals/logs/api-reliability.jsonl is created.
3. Harvester: (Local testing requires gh auth) Run ./scripts/harvest_api_reliability.sh to see it querying GitHub for recent runs.

Validation results of the workflow

• Run of this branch: https://github.com/google-gemini/gemini-cli/actions/runs/23573052212
• Run of this branch + 'chaos' commit on top to simulate errors: https://github.com/google-gemini/gemini-cli/actions/runs/23472109319/job/68296914684

Results of the harvester script

🔍 Fetching runs for 'Testing: E2E (Chained)' created since 2026-03-23 (max 300 runs, branch: alisa/five_hundred_error_api)...
🔍 Fetching runs for 'Evals: Nightly' created since 2026-03-23 (max 300 runs, branch: alisa/five_hundred_error_api)...
could not find any workflows named Evals: Nightly
📭 No runs found for workflow 'Evals: Nightly' since 2026-03-23.

✅ Harvest Complete! Data merged into: api-reliability-summary.jsonl
------------------------------------------------
📊 Gemini API Reliability Summary (Since 2026-03-23)
------------------------------------------------
[
  {
    "model": "gemini-3-pro-preview",
    "500s": 12,
    "503s": 12,
    "retries": 18,
    "skips": 6
  },
  {
    "model": "unknown",
    "500s": 0,
    "503s": 0,
    "retries": 7,
    "skips": 2
  }
]

💡 Total events captured:       33

Pre-Merge Checklist

• Updated relevant documentation and README (if needed)
• Added/updated tests (if needed)
• Noted breaking changes (if any)
• Validated on required platforms/methods:
  • MacOS
    • npm run

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the robustness and observability of evaluation runs by introducing an automatic retry mechanism for transient API errors and a system to collect and analyze these reliability events. It also improves the security of test file handling, ensuring that evaluation tests are more resilient to external API instability and better protected against malicious path manipulations.

Highlights

• API Reliability: Implemented a robust retry mechanism for evaluation tests that encounter transient API errors (500 and 503), retrying up to 3 times before skipping.
• Reliability Logging: Introduced logging for all retry and skip events to evals/logs/api-reliability.jsonl, capturing timestamps, test names, model info, and error codes.
• Reliability Harvester: Added a new shell script (scripts/harvest_api_reliability.sh) that uses the GitHub CLI to download and summarize API reliability logs from GitHub Actions artifacts.
• Security Enhancements: Refactored test file setup to include explicit checks for and block absolute paths or directory traversal attempts in test case definitions.
• CI Configuration: Disabled Vitest's internal retry mechanism for evals and configured artifact uploads for reliability logs in E2E and nightly eval workflows.

Ignored Files

• Ignored by pattern: .github/workflows/** (2)
  • .github/workflows/chained_e2e.yml
  • .github/workflows/evals-nightly.yml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a valuable retry mechanism for transient API errors and a script for harvesting reliability data. The implementation is solid, but I've identified a few areas for improvement concerning asynchronous programming best practices in test-helper.ts and robustness in the harvest_api_reliability.sh script. Specifically, there's a mix of synchronous and asynchronous code that should be made consistent, and the shell script has some potential concurrency issues and misleading documentation.

[github-actions]

Size Change: -4 B (0%)

Total Size: 26.3 MB

Filename	Size	Change
./bundle/chunk-G37HAKQG.js	0 B	-3.64 MB (removed)	🏆
./bundle/chunk-IFPH7QPR.js	0 B	-14.7 MB (removed)	🏆
./bundle/core-FCGFS5E5.js	0 B	-43.6 kB (removed)	🏆
./bundle/devtoolsService-QG3HI6C3.js	0 B	-27.7 kB (removed)	🏆
./bundle/interactiveCli-NGUK7K52.js	0 B	-1.62 MB (removed)	🏆
./bundle/oauth2-provider-IQINTXO6.js	0 B	-9.16 kB (removed)	🏆
./bundle/chunk-433XIV5N.js	14.7 MB	+14.7 MB (new file)	🆕
./bundle/chunk-DR2NOD4F.js	3.64 MB	+3.64 MB (new file)	🆕
./bundle/core-QHVD3BE7.js	43.6 kB	+43.6 kB (new file)	🆕
./bundle/devtoolsService-6NCA47F5.js	27.7 kB	+27.7 kB (new file)	🆕
./bundle/interactiveCli-B66BSVWN.js	1.62 MB	+1.62 MB (new file)	🆕
./bundle/oauth2-provider-EGRBNYMN.js	9.16 kB	+9.16 kB (new file)	🆕

ℹ️ View Unchanged

Filename	Size
./bundle/chunk-34MYV7JD.js	2.45 kB
./bundle/chunk-5AUYMPVF.js	858 B
./bundle/chunk-664ZODQF.js	124 kB
./bundle/chunk-DAHVX5MI.js	206 kB
./bundle/chunk-IUUIT4SU.js	56.5 kB
./bundle/chunk-RJDXJELZ.js	1.96 MB
./bundle/chunk-RJTRUG2J.js	39.8 kB
./bundle/devtools-36NN55EP.js	696 kB
./bundle/dist-T73EYRDX.js	356 B
./bundle/gemini.js	529 kB
./bundle/getMachineId-bsd-TXG52NKR.js	1.55 kB
./bundle/getMachineId-darwin-7OE4DDZ6.js	1.55 kB
./bundle/getMachineId-linux-SHIFKOOX.js	1.34 kB
./bundle/getMachineId-unsupported-5U5DOEYY.js	1.06 kB
./bundle/getMachineId-win-6KLLGOI4.js	1.72 kB
./bundle/memoryDiscovery-5JAQO7MA.js	922 B
./bundle/multipart-parser-KPBZEGQU.js	11.7 kB
./bundle/node_modules/@google/gemini-cli-devtools/dist/client/main.js	221 kB
./bundle/node_modules/@google/gemini-cli-devtools/dist/src/_client-assets.js	227 kB
./bundle/node_modules/@google/gemini-cli-devtools/dist/src/index.js	11.5 kB
./bundle/node_modules/@google/gemini-cli-devtools/dist/src/types.js	132 B
./bundle/sandbox-macos-permissive-open.sb	890 B
./bundle/sandbox-macos-permissive-proxied.sb	1.31 kB
./bundle/sandbox-macos-restrictive-open.sb	3.36 kB
./bundle/sandbox-macos-restrictive-proxied.sb	3.56 kB
./bundle/sandbox-macos-strict-open.sb	4.82 kB
./bundle/sandbox-macos-strict-proxied.sb	5.02 kB
./bundle/src-QVCVGIUX.js	47 kB
./bundle/tree-sitter-7U6MW5PS.js	274 kB
./bundle/tree-sitter-bash-34ZGLXVX.js	1.84 MB

_{compressed-size-action}

[gemini-code-assist]

Code Review

This pull request introduces a retry mechanism for transient API errors (500/503) in evaluation tests and adds a script to harvest reliability data. A security audit found no high or critical severity vulnerabilities, and the security enhancements to prevent path traversal in test file setup are a positive improvement. The changes are well-implemented with good test coverage, and the suggestion to improve error handling in the new harvester script to make it more robust against failures like authentication issues is valid.