feat(plan): add behavioral evals for plan mode

[jerop]

Adds behavioral evaluations to verify that the agent correctly adheres to its restrictions in Plan Mode, including refusing file modifications and correctly entering/exiting the mode.

Closes #17169

[gemini-code-assist]

Summary of Changes

Hello @jerop, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the robustness of the agent's "Plan Mode" by adding comprehensive behavioral tests. These evaluations are designed to confirm that the agent strictly adheres to its operational constraints within this mode, particularly its read-only nature for file systems, and that it transitions into and out of Plan Mode as intended by user prompts.

Highlights

• New Behavioral Evaluations: Introduces a suite of behavioral evaluations specifically for the agent's "Plan Mode" functionality.
• Restriction Adherence: Verifies that the agent correctly refuses file modification attempts while operating in Plan Mode.
• Mode Transitions: Ensures the agent properly enters Plan Mode when prompted to create a plan and exits Plan Mode when implementation is requested after a plan is complete.

Changelog

• evals/plan_mode.eval.ts
  • Added a new evaluation file to test the agent's "Plan Mode" behavior.
  • Includes a test to ensure the agent refuses file modification attempts while in plan mode.
  • Adds a test to verify the agent correctly enters plan mode when asked to create a plan.
  • Incorporates a test to confirm the agent exits plan mode when a plan is complete and implementation is requested.

Activity

• Pull request created by jerop with the title "feat(plan): add behavioral evals for plan mode".
• The purpose is to add behavioral evaluations to verify agent adherence to Plan Mode restrictions, including refusing file modifications and correct mode entry/exit.
• This PR addresses and closes issue Add behavioral evaluations for planning tools and workflow #17169.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request adds valuable behavioral evaluations for Plan Mode. The tests cover key scenarios: refusing file modifications, entering plan mode, and exiting plan mode. I've found a critical issue in one of the test assertions that would cause it to fail due to a type error. My suggestion corrects this, ensuring the test validates the intended behavior correctly.

[github-actions]

Size Change: -2 B (0%)

Total Size: 23.8 MB

ℹ️ View Unchanged

Filename	Size	Change
./bundle/gemini.js	23.7 MB	-2 B (0%)
./bundle/sandbox-macos-permissive-closed.sb	1.03 kB	0 B
./bundle/sandbox-macos-permissive-open.sb	890 B	0 B
./bundle/sandbox-macos-permissive-proxied.sb	1.31 kB	0 B
./bundle/sandbox-macos-restrictive-closed.sb	3.29 kB	0 B
./bundle/sandbox-macos-restrictive-open.sb	3.36 kB	0 B
./bundle/sandbox-macos-restrictive-proxied.sb	3.56 kB	0 B

_{compressed-size-action}

[jerop]

/gemini review

[gemini-code-assist]

Code Review

This pull request adds valuable behavioral evaluations for the new Plan Mode feature. The tests cover entering and exiting plan mode, and correctly refusing file modifications for non-plan files. My review focuses on improving the completeness of these evaluations. I've identified a critical missing test case for allowed file modifications within the plans directory. Adding this test is important not only for comprehensive coverage but also because it may reveal a potential bug in how file modification policies are currently enforced in plan mode. I've provided a detailed comment with a code suggestion to add this test.

[gundermanc]

If you haven't, I'd recommend running this with the actions run: https://github.com/google-gemini/gemini-cli/actions/workflows/evals-nightly.yml

It will run each test 3X and on each of the different models and generate a report.