Skip to content

fix: aggregate probes details in digest#1459

Merged
jmartin-tech merged 2 commits intoNVIDIA:mainfrom
parkanzky:fix/aggregrate-probespecs
Nov 6, 2025
Merged

fix: aggregate probes details in digest#1459
jmartin-tech merged 2 commits intoNVIDIA:mainfrom
parkanzky:fix/aggregrate-probespecs

Conversation

@parkanzky
Copy link
Collaborator

When aggregating jsonl reports from multiple garak invocations using garak.analyze.aggregate_reports, probe specs in the start_run and digest entries were not correctly aggregated. This change aggregates the probe specs from each jsonl output.

Tests had included the incorrect digest aggregation and ignored incorrect start_run aggregation. This updates the test and tests both places.

Verification

List the steps needed to make sure this thing works

  • Run the tests and ensure they pass python -m pytest tests/
  • Run garak several times with different probespecs
  • Aggregate output jsonls - python -m garak.analyze.aggregate_reports -o ./aggregated.jsonl <list of report jsonls>
  • Verify that the start_run entry plugins.probe_spec string value is a comma separated list of aggregated probespecs which includes all probespecs from all aggregated report jsonls.
  • Verify that the final digest entry probespec string value is the same comma separated list of aggregated probespecs
  • Generate the HTML report from the aggregated jsonl - python -m garak.analyze.report_digest -w -r ./aggregated.jsonl > ./report.html
  • Verify that the HTML config dropdown probe spec: value includes the aggregated probespecs from individual runs.

Signed-off-by: Paul A. Parkanzky <pparkanzky@nvidia.com>
@parkanzky parkanzky self-assigned this Nov 5, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 5, 2025

DCO Assistant Lite bot All contributors have signed the DCO ✍️ ✅

@jmartin-tech jmartin-tech changed the title fix(analyze): aggregate probes in digest fix: aggregate probes details in digest Nov 5, 2025
@parkanzky
Copy link
Collaborator Author

recheck

@parkanzky
Copy link
Collaborator Author

I have read the DCO Document and I hereby sign the DCO

@parkanzky
Copy link
Collaborator Author

recheck

github-actions bot added a commit that referenced this pull request Nov 5, 2025
@jmartin-tech jmartin-tech self-assigned this Nov 5, 2025
Signed-off-by: Paul A. Parkanzky <pparkanzky@nvidia.com>
Copy link
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this revision now builds the digest instead of making an attempt to merge all the various entires that might exist it results in calculations for z_scores and comments that based on current the installed version's calibration data. I am good with that change as aggregation should be performed using the same version that generated the individual reports, @leondz can you weigh in on if this is an acceptable change.

@jmartin-tech jmartin-tech dismissed their stale review November 6, 2025 17:02

Changes accounted for, further review in progress.

@jmartin-tech jmartin-tech merged commit e2e9529 into NVIDIA:main Nov 6, 2025
15 checks passed
@github-actions github-actions bot locked and limited conversation to collaborators Nov 6, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants