-
Notifications
You must be signed in to change notification settings - Fork 10
Description
Problem
The docs site has strong reference documentation for agent tracing (Agent Traces UI, Agent Graph & Path, Python SDK, Agent Health), but the onboarding flow for an agent developer — someone building an LLM-powered agent who wants to trace and evaluate it — has gaps that make it harder to reach the "aha moment."
This is a companion to #112 (APM persona). Together they cover the two primary user personas for the platform.
Persona: Agent Developer
Who: Building or operating an LLM-powered agent (chatbot, autonomous agent, multi-agent system). Uses frameworks like Strands, LangChain, or raw OpenAI/Anthropic/Bedrock SDKs.
Goal: Instrument their agent → see agent traces in OSD → evaluate agent quality with Agent Health.
Audit Findings
1. No "Instrument your agent" quickstart
This is the biggest gap. An agent developer wants: "I have a Python agent. How do I add tracing and see it in Agent Traces?"
The current paths are fragmented:
- Landing page → "Trace an AI Agent" → links to
/docs/ai-observability/agent-tracing/→ this is the OSD UI reference, not instrumentation guidance - SDKs → Python SDK → solid API reference for
opensearch-genai-sdk-py, but not a guided tutorial - Send Data → Applications → Python → generic OTel instrumentation, not agent-specific
There is no guided tutorial that connects: "pip install SDK → add decorators → run agent → see trace in Agent Traces UI."
2. "Trace an AI Agent" quickstart links to the wrong page
The landing page quickstart card "Trace an AI Agent" links to /docs/ai-observability/agent-tracing/ — the OSD UI reference page. An agent developer clicking this expects to learn how to instrument their agent, not how to navigate the UI. The page opens with architecture diagrams and PPL queries, not pip install.
3. Fragmented instrumentation guidance
An agent developer has three places to look, none of which cross-link:
| Page | Content | Missing |
|---|---|---|
| SDKs → Python SDK | opensearch-genai-sdk-py decorators (@agent, @tool, @workflow) |
No link to Agent Traces UI or Agent Health |
| Send Data → Python | Generic OTel with manual gen_ai.* attributes |
No mention of the SDK |
| AI Observability → Agent Tracing | OSD UI reference, mentions GenAI conventions | No link to SDK or how to emit the right attributes |
4. No bridge from Agent Traces → Agent Health
These are separate sidebar sections with no cross-linking. An agent developer viewing traces in OSD has a natural next question: "How do I evaluate whether my agent is doing the right thing?" There is no callout or link from Agent Traces → Agent Health.
The relationship between the two is also unclear — Agent Health has its own trace visualization that overlaps with OSD Agent Traces. When should a user use which?
5. No framework-specific agent instrumentation guides
Agent developers think in frameworks: "I use Strands," "I use LangChain," "I use CrewAI." The repo has examples/strands/ and examples/langchain/ directories, and the README mentions StrandsTelemetry, but none of this is linked from the docs site.
6. Sidebar label formatting
The sidebar shows agent-tracing (auto-generated from directory name) instead of "Agent Tracing." This is the only sidebar item with this formatting.
What Works Well
- Agent Health onboarding is genuinely good —
npxone-liner with demo data, step-by-step Getting Started with UI and CLI options, screenshots - Agent Traces reference docs are thorough — span categories, PPL queries, flyout details, DAG visualization
- Python SDK API reference is comprehensive —
register(), decorators,score() - Landing page correctly prioritizes the agent persona with a dedicated quickstart card
Proposed Changes
P0 — New quickstart: "Instrument and trace your first agent"
- 5-minute tutorial: install SDK, add decorators to a simple agent, run it, see the trace in OSD Agent Traces
- Screenshots of the Agent Traces UI showing the resulting trace
- Link to Agent Health as the natural next step
P0 — Fix "Trace an AI Agent" landing page link
- Should point to the new instrumentation quickstart, not the UI reference page
P1 — Cross-link SDK ↔ Agent Traces ↔ Agent Health
- Python SDK page → "See your traces in Agent Traces"
- Agent Traces page → "Instrument your agent with the Python SDK"
- Agent Traces page → "Evaluate agent quality with Agent Health"
- Agent Health page → "View production traces in Agent Traces"
P1 — Add framework-specific guides or links
- At minimum, link to the existing
examples/strands/andexamples/langchain/directories - Ideally, short guides for Strands (
StrandsTelemetry) and LangChain instrumentation
P2 — Fix sidebar label
agent-tracing→ "Agent Tracing" (configure inastro.config.mjs)
P2 — Add "Agent Traces vs Agent Health" guidance
- Short section explaining when to use OSD Agent Traces (production monitoring, trace exploration) vs Agent Health (evaluation, golden path comparison, batch experiments)
Context
Companion to #112 (APM persona onboarding). Together these two issues cover the primary user personas for the platform.