Skip to content

[DOCS] Improve onboarding flow for Agent Developer persona #113

@kylehounslow

Description

@kylehounslow

Problem

The docs site has strong reference documentation for agent tracing (Agent Traces UI, Agent Graph & Path, Python SDK, Agent Health), but the onboarding flow for an agent developer — someone building an LLM-powered agent who wants to trace and evaluate it — has gaps that make it harder to reach the "aha moment."

This is a companion to #112 (APM persona). Together they cover the two primary user personas for the platform.

Persona: Agent Developer

Who: Building or operating an LLM-powered agent (chatbot, autonomous agent, multi-agent system). Uses frameworks like Strands, LangChain, or raw OpenAI/Anthropic/Bedrock SDKs.

Goal: Instrument their agent → see agent traces in OSD → evaluate agent quality with Agent Health.

Audit Findings

1. No "Instrument your agent" quickstart

This is the biggest gap. An agent developer wants: "I have a Python agent. How do I add tracing and see it in Agent Traces?"

The current paths are fragmented:

  • Landing page → "Trace an AI Agent" → links to /docs/ai-observability/agent-tracing/ → this is the OSD UI reference, not instrumentation guidance
  • SDKs → Python SDK → solid API reference for opensearch-genai-sdk-py, but not a guided tutorial
  • Send Data → Applications → Python → generic OTel instrumentation, not agent-specific

There is no guided tutorial that connects: "pip install SDK → add decorators → run agent → see trace in Agent Traces UI."

2. "Trace an AI Agent" quickstart links to the wrong page

The landing page quickstart card "Trace an AI Agent" links to /docs/ai-observability/agent-tracing/ — the OSD UI reference page. An agent developer clicking this expects to learn how to instrument their agent, not how to navigate the UI. The page opens with architecture diagrams and PPL queries, not pip install.

3. Fragmented instrumentation guidance

An agent developer has three places to look, none of which cross-link:

Page Content Missing
SDKs → Python SDK opensearch-genai-sdk-py decorators (@agent, @tool, @workflow) No link to Agent Traces UI or Agent Health
Send Data → Python Generic OTel with manual gen_ai.* attributes No mention of the SDK
AI Observability → Agent Tracing OSD UI reference, mentions GenAI conventions No link to SDK or how to emit the right attributes

4. No bridge from Agent Traces → Agent Health

These are separate sidebar sections with no cross-linking. An agent developer viewing traces in OSD has a natural next question: "How do I evaluate whether my agent is doing the right thing?" There is no callout or link from Agent Traces → Agent Health.

The relationship between the two is also unclear — Agent Health has its own trace visualization that overlaps with OSD Agent Traces. When should a user use which?

5. No framework-specific agent instrumentation guides

Agent developers think in frameworks: "I use Strands," "I use LangChain," "I use CrewAI." The repo has examples/strands/ and examples/langchain/ directories, and the README mentions StrandsTelemetry, but none of this is linked from the docs site.

6. Sidebar label formatting

The sidebar shows agent-tracing (auto-generated from directory name) instead of "Agent Tracing." This is the only sidebar item with this formatting.

What Works Well

  • Agent Health onboarding is genuinely good — npx one-liner with demo data, step-by-step Getting Started with UI and CLI options, screenshots
  • Agent Traces reference docs are thorough — span categories, PPL queries, flyout details, DAG visualization
  • Python SDK API reference is comprehensive — register(), decorators, score()
  • Landing page correctly prioritizes the agent persona with a dedicated quickstart card

Proposed Changes

P0 — New quickstart: "Instrument and trace your first agent"

  • 5-minute tutorial: install SDK, add decorators to a simple agent, run it, see the trace in OSD Agent Traces
  • Screenshots of the Agent Traces UI showing the resulting trace
  • Link to Agent Health as the natural next step

P0 — Fix "Trace an AI Agent" landing page link

  • Should point to the new instrumentation quickstart, not the UI reference page

P1 — Cross-link SDK ↔ Agent Traces ↔ Agent Health

  • Python SDK page → "See your traces in Agent Traces"
  • Agent Traces page → "Instrument your agent with the Python SDK"
  • Agent Traces page → "Evaluate agent quality with Agent Health"
  • Agent Health page → "View production traces in Agent Traces"

P1 — Add framework-specific guides or links

  • At minimum, link to the existing examples/strands/ and examples/langchain/ directories
  • Ideally, short guides for Strands (StrandsTelemetry) and LangChain instrumentation

P2 — Fix sidebar label

  • agent-tracing → "Agent Tracing" (configure in astro.config.mjs)

P2 — Add "Agent Traces vs Agent Health" guidance

  • Short section explaining when to use OSD Agent Traces (production monitoring, trace exploration) vs Agent Health (evaluation, golden path comparison, batch experiments)

Context

Companion to #112 (APM persona onboarding). Together these two issues cover the primary user personas for the platform.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationuntriagedIssues that have not been triaged

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions