Interview Metrics & Analytics

[mathaix]

User Story

As a manager
I want to view metrics about interview performance and costs
So that I can optimize future discovery initiatives

Acceptance Criteria

Interview-Level Metrics

For each interview:

• Duration: Total time from start to completion
• Turn Count: Number of messages exchanged
• Topic Coverage: Percentage of planned topics discussed
• Entity Count: Number of entities extracted
• Confidence: Average confidence of extracted entities
• Cost: Total LLM tokens used × cost per token
• Model Used: Which Claude model (Haiku, Sonnet, Opus)

Project-Level Metrics

Aggregated across all interviews:

• Total Interviews: Invited, completed, in-progress, cancelled
• Completion Rate: Completed / invited
• Average Duration: Mean interview length
• Total Cost: Sum of all interview costs
• Cost per Interview: Average
• Entity Extraction Rate: Entities per interview
• Coverage Achievement: % of topics covered across project
• Time to Completion: Invite to complete duration

Cost Breakdown

• By model: Haiku vs Sonnet vs Opus usage
• Input tokens vs output tokens
• By agent: Which agents are most expensive
• Trend over time: Cost per day/week

Performance Analytics

• Interview duration distribution (histogram)
• Topic coverage distribution
• Entity extraction trends over time
• Completion rate by agent
• Time-of-day patterns

Comparison Metrics

• Compare to project goals (from blueprint)
• Compare to organizational benchmarks
• Compare to previous projects

Visualizations

• Cost chart (line graph over time)
• Coverage heat map (topics × interviews)
• Duration box plot
• Entity type distribution (pie chart)
• Interview funnel (invited → completed)

Export Analytics

• Download metrics as CSV
• Include in synthesis report
• API endpoint for programmatic access

Technical Notes

class InterviewMetrics(BaseModel):
    interview_id: str
    duration_seconds: int
    turn_count: int
    topic_coverage_pct: float
    entity_count: int
    avg_confidence: float
    input_tokens: int
    output_tokens: int
    cost_usd: Decimal
    model: str
    
class ProjectMetrics(BaseModel):
    project_id: str
    total_interviews: int
    completion_rate: float
    avg_duration_seconds: int
    total_cost_usd: Decimal
    cost_per_interview_usd: Decimal
    entity_extraction_rate: float
    coverage_achievement_pct: float

Store metrics in PostgreSQL with time-series queries. Use Logfire for real-time cost tracking.

Definition of Done

• Interview-level metrics calculated
• Project-level metrics aggregated
• Cost tracking working
• Visualizations rendering
• Export functionality working
• Code reviewed and merged

Requirement ID

RS-08

---

🤖 Generated with Claude Code