fix: extract A2A_INTERACTION content in categorical evaluation transcripts#41
Closed
evekhm wants to merge 1 commit intoGoogleCloudPlatform:mainfrom
Closed
fix: extract A2A_INTERACTION content in categorical evaluation transcripts#41evekhm wants to merge 1 commit intoGoogleCloudPlatform:mainfrom
evekhm wants to merge 1 commit intoGoogleCloudPlatform:mainfrom
Conversation
…ripts Add JSON_VALUE(content, '$.artifacts[0].parts[0].text') to the COALESCE chain in all 4 transcript-building SQL queries. A2A_INTERACTION events (from ADK's RemoteA2aAgent) store the remote agent's response under artifacts[0].parts[0].text, which was not being extracted — causing AI.GENERATE to see a question with no answer and return NULL for ~40% of sessions. Fixes GoogleCloudPlatform#40 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Collaborator
|
drop the PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JSON_VALUE(content, '$.artifacts[0].parts[0].text')to the COALESCE chain in all 4 transcript-building SQL queries incategorical_evaluator.pyA2A_INTERACTIONevents (from ADK'sRemoteA2aAgentviaTransferToAgentTool) store the remote agent's response underartifacts[0].parts[0].text, which was not being extractedDetails
The transcript-building SQL uses
COALESCEto extract text from event content:However,
A2A_INTERACTIONevents (introduced in google/adk-python#5325) store the remote agent's response under a different JSON structure:{ "artifacts": [{"parts": [{"kind": "text", "text": "The actual agent response..."}]}], "contextId": "...", "history": [...] }None of the existing JSON paths match. The fix adds
JSON_VALUE(content, '$.artifacts[0].parts[0].text')before the$.toolfallback in all 4 locations:CATEGORICAL_TRANSCRIPT_QUERY(line 233)CATEGORICAL_AI_GENERATE_QUERY(line 259)build_ai_classify_query()(line 380)build_ai_generate_query()(line 449)Fixes #40
Test plan
evaluate_categorical()on a dataset containing A2A sessions (usingRemoteA2aAgent)🤖 Generated with Claude Code