How to properly track external service costs in LangChain tools with LangFuse?

[ivishalanand]

Problem Statement

I'm trying to implement cost tracking for external services (Tavily web search, Pinecone vector DB, OpenAI embeddings, Cohere rerank) in LangFuse traces, but despite successful API calls and "success" logs, the cost data never appears in the LangFuse UI.

Current behavior:

• External service cost calculations work correctly (e.g., $0.016 for Tavily search)
• LangFuse tool observations are created successfully
• LLM costs (ChatAnthropic) track perfectly and appear in UI
• External service costs show as empty: costDetails: {}, usage: {input: 0, output: 0}

Setup Context

• Architecture: LangChain tools (@tool decorator) → LangGraph ToolNode → LangFuse CallbackHandler
• SDK Version: LangFuse Python SDK v4.x
• Integration: LangFuse CallbackHandler attached at request scope
• Tool execution: External service calls happen within async tool functions

What We've Tried

Approach 1: update_current_generation()

@tool("web-search")
async def web_search(query: str):
    # External service call
    results, cost_details, usage_details = await external_service(query)
    
    # Cost attribution attempt
    get_client().update_current_generation(
        cost_details=cost_details,  # {"input": 0.016}
        usage_details=usage_details  # {"input": 2, "queries": 1}
    )
    return results

Result: Logs show success, but LangFuse API shows costDetails: {}

Approach 2: Manual generation creation

@tool("web-search")  
async def web_search(query: str):
    results, cost_details, usage_details = await external_service(query)
    
    # v4 pattern attempt
    client = get_client()
    generation = client.generation(
        name="external-service-cost",
        input={"query": query},
        model="tavily-advanced",
        cost_details=cost_details,
        usage_details=usage_details
    )
    generation.end()
    return results

Result: No errors, but cost generation doesn't appear in trace

Approach 3: Context manager pattern

with client.start_as_current_observation(
    name="external_cost_tracking",
    as_type="generation"
) as obs:
    obs.update(
        cost_details=cost_details,
        usage_details=usage_details
    )

Result: Same issue - costs don't persist

Evidence from LangFuse API

Direct API query of trace shows:

{
  "name": "web-search",
  "type": "TOOL", 
  "cost": null,
  "costDetails": {},  // ❌ Should contain external costs
  "usage": {"input": 0, "output": 0, "total": 0}  // ❌ Should show usage
}

Meanwhile, LLM observations work perfectly:

{
  "name": "ChatAnthropic",
  "type": "GENERATION",
  "costDetails": {"total": 0.031646},  // ✅ Works fine
  "usage": {"input": 8084, "output": 67}
}

Questions for the Community

1. Best practices: What's the recommended approach for tracking external service costs in LangChain tools?

2. Timing issues: Should cost attribution happen during tool execution, or after completion?

3. Context management: How do we ensure cost updates target the correct observation when using LangGraph + LangChain tools + async operations?

4. Known limitations: Is this a known issue with the current LangFuse + LangChain integration?

5. Alternative approaches: Should we implement cost tracking at the LangGraph level instead of individual tools?

Why This Matters

This affects common LLM application patterns:

• RAG pipelines: Need to track embedding + vector DB + rerank costs
• Web search: Track search API costs alongside LLM costs
• Multi-service tools: Any tool using multiple paid APIs
• Cost optimization: Understanding true cost per request for optimization

Additional Context

• All external service integrations work correctly (results appear in responses)
• Cost calculation logic is verified and accurate
• LangFuse tracing works for LLM calls
• The issue seems to be specifically with persisting manually calculated costs

Any guidance on proper implementation patterns would be greatly appreciated! Happy to contribute documentation once we find the solution.

---

Environment:

• Python 3.12
• LangFuse Python SDK 4.x
• LangChain + LangGraph
• FastAPI backend

[inkeep-ai-assistant[bot]]

Based on the available documentation, I can provide guidance on tracking external service costs in Langfuse, though the sources don't cover the specific LangChain tool integration pattern you're describing.

Recommended Approach: Manual Generation with Usage and Cost Details

For tracking external service costs, you should create observations of type generation and pass usage_details and cost_details directly. Here's the pattern from the documentation:
(1)

from langfuse import get_client

langfuse = get_client()

with langfuse.start_as_current_observation(
    as_type="generation",
    name="openai-style-generation",
    model="gpt-4o"
) as generation:
    # Simulate LLM call
    # response = openai_client.chat.completions.create(...)

    generation.update(
        usage_details={
            # usage (OpenAI-style schema)
            "prompt_tokens": 10,
            "completion_tokens": 25,
            "total_tokens": 35,
            "prompt_tokens_details": {
                "cached_tokens": 5,
                "audio_tokens": 2,
            },
            "completion_tokens_details": {
                "reasoning_tokens": 15,
            },
        }
    )

(1)

For custom cost details, you can pass them manually:
(1)

generation.update({
    usageDetails: {
        input: 10,
        output: 5,
        cache_read_input_tokens: 2,
        some_other_token_count: 10,
        total: 17,  # optional, it is derived from input + cache_read_input_tokens + output
    },
    costDetails: {
        # If you don't want the costs to be calculated based on model definitions, you can pass the costDetails manually.
        input: 1,
        output: 1,
        cache_read_input_tokens: 0.5,
        some_other_token_count: 1,
        total: 3.5,
    },
})

(1)

Important Notes

• Only generation and embedding observation types can track costs and usage. Your external service calls need to be wrapped as generations, not as generic spans or tool observations.
  (1)

• Ensure you call langfuse.flush() before your application exits to ensure all traces are sent, especially in serverless or short-lived contexts.
  (2)

Potential Issue: Context and Parent-Child Relationships

When using the LangChain CallbackHandler alongside manual Langfuse SDK calls, there may be context isolation issues. The documentation notes that filtering or context management can break parent-child relationships in traces.
(3)

Limitations of Available Documentation

The sources don't specifically cover:

• How to inject cost tracking into LangChain @tool decorated functions when using the CallbackHandler
• How LangGraph's ToolNode interacts with manual Langfuse observations
• Whether update_current_generation() works within LangChain tool contexts

Check out these potentially useful sources for more context:

• Model Usage & Cost Tracking
• Advanced features

Would you like me to provide a general answer based on my broader knowledge of how these integrations typically work?

📚 Sources:

1. Model Usage & Cost Tracking
2. bug: tracing issues in latest cloud release (v3.113.0)
3. Advanced features

---

Have another question? Just tag @inkeep.