Skip to content

bug: coder template LLM writes own tests instead of using existing ones #345

@qhkm

Description

@qhkm

Problem

When running zeptoclaw agent --template coder -m "Read file X, find bugs, fix them, verify with tests", the LLM:

  1. Reads the buggy file correctly
  2. Writes its own test suite that matches the buggy behavior (asserting wrong expected values)
  3. Runs its own tests — which pass because they test the wrong things
  4. Reports "NO BUGS FOUND" / "ALL TESTS PASSED"
  5. Never actually edits the original file

Expected

The agent should:

  • Use the existing test_calculator.py to verify (run it first to see failures)
  • Edit the original file in-place using edit_file
  • Re-run the existing tests to confirm the fix

Root cause

The coder template system prompt doesn't instruct the LLM to:

  • Check for existing test files before writing new ones
  • Use edit_file to fix bugs in-place rather than write_file to create new files
  • Run existing tests as the verification step

Reproduction

cp -R test-coding-pristine test-coding-zepto
./target/release/zeptoclaw agent --template coder -m "Read the file test-coding-zepto/calculator.py, find all the bugs, and fix them. Show me what you changed, and verify the result with tests."
python3 -m unittest test-coding-zepto/test_calculator.py -v  # still 6/6 FAIL

Suggested fix

Update the coder template system prompt to include instructions like:

  • "Always check for existing test files first and run them to understand the current state"
  • "Fix bugs by editing the original file with edit_file, not writing new files"
  • "Verify by running the existing test suite, not by writing your own tests"

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2-highImportant, next uparea:cliCLI commands (src/cli/)bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions