[OPIK-5655] [Evaluation Suite] Evaluation suite task should support returning just the agent output

[yaricom]

Details

Current behavior:

When using evaluation suites, the task needs to return a dictionary with an input and output key so that we can score the assertion:

def evaluation_task(data):
    response = call_llm(data)
    return {"output": response, "input": data}


eval_suite.run(
    task=evaluation_task,
)

Expected behavior

I expected to simply return the output of my agent in the evaluation task and to have that scored against the assertion. We could simplify the experience here by passing the evaluation suite data field as the input to the LLM as a Judge metric and the task output as the output.

The SDK method would become:

def evaluation_task(data):
    return call_llm(data)


eval_suite.run(
    task=evaluation_task,
)

---

Summary

Relaxes the return-type contract for EvaluationSuite task functions: instead of requiring an explicit {"input": ..., "output": ...} dict, any non-dict return value is now automatically wrapped, and a pinned anthropic version is added to fix a CrewAI integration test dependency error.

Changes by Component

Python SDK

• validate_task_result now accepts any return type — non-dict values are auto-wrapped as {"output": result}; when called with input_data, the wrapper becomes {"input": input_data, "output": result}.
• Dict results are still validated to require both "input" and "output" keys, raising ValueError if either is missing.
• _validated_task in __internal_api__run_optimization_suite__ now passes the item data dict as input_data so auto-wrapped results carry the task input automatically.
• Updated run() and __internal_api__run_optimization_suite__ docstrings and examples to document both the simplified and explicit return styles.
• Replaced the stale test_non_dict__raises_type_error test with correct auto-wrapping assertions; added 6 new unit tests covering all wrapping scenarios (string, int, None, list, with/without input_data).
• Added anthropic>=0.88.0 to tests/library_integration/crewai/requirements_v1.txt to fix AttributeError: 'OpenAICompletion' object has no attribute 'client' in CrewAI integration tests.

Files Changed

 sdks/python/src/opik/api_objects/dataset/evaluation_suite/evaluation_suite.py | 75 ++++++++++++++++------
 sdks/python/tests/library_integration/crewai/requirements_v1.txt              |  1 +
 sdks/python/tests/unit/api_objects/dataset/evaluation_suite/test_evaluation_suite.py | 53 ++++++++++++---
 3 files changed, 98 insertions(+), 31 deletions(-)

Change checklist

• User facing
• Documentation update

Issues

• Resolves #
• OPIK-5655

AI-WATERMARK

AI-WATERMARK: yes

• If yes:
  • Tools: Claude Code v2.1.81
  • Model(s): Sonnet 4.6
  • Scope: Tests
  • Human verification: Done

Testing

Added related unit tests

Documentation

Updated docstring