expand items in report entry_type:eval#1547
Merged
leondz merged 4 commits intoNVIDIA:mainfrom Jan 15, 2026
Merged
Conversation
jmartin-tech
approved these changes
Jan 15, 2026
Collaborator
jmartin-tech
left a comment
There was a problem hiding this comment.
Testing looks good. This revision reflects a breaking change that impacts report aggregation and html generation for reports generated on version prior to the change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds counts for skipped and failed evals, and relays totals for both processed and evaluated output counts.
Previously the report
evalentry only listed one "total" and it was ambiguous whether or not this was withNoneoutputs, leading to unstable counting.Now this is clarified:
passed- number of passing outputsfails- number of failing outputs (hits)nones- number ofNones from generator/detectortotal_processed- total number of results from the generator/probe processed and passed to the detectortotal_evaluated- total number of target outputs evaluated (for most detectors, this will excludeNones)