LLM-as-a-Judge Evaluator not running on live traces

[gadirajupavantiwari]

Describe your question

why no live incoming traces are evaluated by llm as judge even it is active and 100% sampling rate there. logs page in UI shows empty.

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

Several common reasons can cause LLM-as-a-Judge evaluators to not run on live traces even when active with 100% sampling:

1. Variable Mapping Issues
All mapped variables must exist in every trace for evaluation to succeed. If any required variables (like {{input}}, {{output}}) are missing from your traces, jobs will be skipped.(1)

2. SDK Version Requirements
For observation-level evaluators, you need Python SDK v3+ or JS/TS SDK v4+.(1)

For trace-level evaluators with OTel-based SDKs (Python v3+ or JS/TS v4+), trace input/output is derived from the root observation by default. To explicitly set trace input/output, use set_trace_io() (Python) or setTraceIO() (JS/TS).(1)

3. Attribute Propagation for Observation-Level Evaluators
If you're using observation-level evaluators and filtering by trace-level attributes (userId, sessionId, tags, metadata, traceName), you must use propagate_attributes() in your instrumentation code. Without this, trace attributes won't be available on observations, and your observations won't be matched by the evaluator.(1)

4. Filter Configuration
If your filters are too restrictive or required variables are missing in traces, no evaluation jobs will be created. Preview matched traces from the last 24 hours to validate your filter configuration.(1)

Recommended Steps:

• Check that your traces contain the data mapped to evaluator variables
• Verify your SDK version meets requirements
• If using observation-level evaluators with trace filters, ensure you're propagating attributes
• Review the evaluator logs for any errors or pending jobs

Check out these potentially useful sources for more troubleshooting: Troubleshooting and FAQ, Why is my observation-level evaluator not executing?(2)

📚 Sources:

1. LLM-as-a-Judge
2. Troubleshooting and FAQ

---

Have another question? Just tag @inkeep.

[Lotte-Verheyden]

Hi @gadirajupavantiwari, this is often because users are using an older SDK version. The observation-level LLM-as-a-Judge feature only works for Python v3+ or JS/TS v4+, see https://langfuse.com/faq/all/llm-as-a-judge-migration#prerequisites