(feat): Add a ToDo tool to track ongoing task lists

[anj-s]

TLDR

This PR introduces a new write_todos_list tool that allows the agent to create and manage a checklist of tasks for complex user requests. This helps the agent track its progress, organize its work, and provides the user with visibility into the agent's plan.

Dive Deeper

The write_todos_list tool is a declarative tool that enables the agent to manage a list of tasks with the following statuses: pending, in_progress, completed, and cancelled. The agent is guided by an updated system prompt on when and how to use this tool, with a focus on using it for complex, multi-step tasks and avoiding it for simple requests.

The tool is enabled by a useWriteTodos flag in the configuration. The implementation includes the tool itself, along with comprehensive unit tests to ensure its functionality and validation logic are working correctly.

Reviewer Test Plan

To test this feature, you can enable the useWriteTodos flag in your settings and give the agent a complex task. Here are a few examples:

1. Create a new feature:

   • Prompt: add a new feature to the CLI that allows users to configure the output format of the response.
   • Expected behavior: The agent should create a todo list with steps like add a new configuration option, implement the logic to format the output, add tests for the new feature, etc.

2. Build a simple application:

   • Prompt: create a simple web app that uses the Gemini API to answer questions.
   • Expected behavior: The agent should break down the task into smaller sub-tasks and create a todo list to track its progress.

3. Debug an issue:

   • Prompt: The application is crashing when I try to upload a file. Can you help me debug and fix the issue?
   • Expected behavior: The agent should create a todo list to investigate the issue, such as reproduce the crash, examine the logs, identify the root cause, implement a fix, and verify the fix.

Fixes #4580

Testing Matrix

	🍏	🪟	🐧
npm run	❓	❓	❓
npx	❓	❓	❓
Docker	❓	❓	❓
Podman	❓	-	-
Seatbelt	❓	-	-

Linked issues / bugs

[anj-s]

I'm confused. All the tool does is respond back with the todos it was given. It doesn't persist it anywhere and doesn't allow the model to view the list. Is this going to be part of a suite of tools?

This is a tool for the model to track what it needs to do. The todos are provided by the model and this tool is for it to communicate the updated status of that list. It doesn't need to persist it anywhere since it is in the history. Yes, this will be part of the core tools.

[scidomino]

It's weird that a noop tool would improve performance but I assume you have run evals and shown that this improves things.

[owenofbrien]

It's weird that a noop tool would improve performance but I assume you have run evals and shown that this improves things.

I think it's primarily a way to:
a) encourage the model to actually make a plan for complex tasks, and
b) encourage the model to explicitly update the plan as tasks are completed or become obsolete

I wonder if just adding instructions for a) and b) to the system prompt could yield a similar performance impact. @anj-s wdyt?

[anj-s]

It's weird that a noop tool would improve performance but I assume you have run evals and shown that this improves things.

Its not a noop tool as explained above. This helps the model create a list of items and track it. yes, this improves evals and is a known method for doing so.

[anj-s]

It's weird that a noop tool would improve performance but I assume you have run evals and shown that this improves things.

I think it's primarily a way to: a) encourage the model to actually make a plan for complex tasks, and b) encourage the model to explicitly update the plan as tasks are completed or become obsolete

I wonder if just adding instructions for a) and b) to the system prompt could yield a similar performance impact. @anj-s wdyt?

We have this in the system prompt but its not something the model does consistently and does not involve the model updating the plan list at every turn. We ideally want the todo list to be the only plan list that the model is tracking