script: qualitative review output#1144
Conversation
leondz
added
the
reporting
|
A general thought here, would there be value in the |
Yeah, this update has made my to-do list a few times but never been actioned. I think this script is currently the only thing that would consume it, but as reporting improves and tooling around it becomes more diverse, this change will have more impact. |
|
noted a condition where not all probe results are reported |
|
Is the absolute threshold at Tier 1 too easy? I feel all three of these are actually not so great and are worthy of documenting:
tagging @erickgalinkin for extra input |
garak/analyze/__init__.py
Outdated
| # SPDX-FileCopyrightText: Portions Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| SCORE_DEFCON_BOUNDS = [0.05, 0.4, 0.8, 0.99] |
There was a problem hiding this comment.
Better to cast this as a typing.Dict[str, float] and then map each tier to it's bound? Prevents look up issues later with having to get the order correct.
There was a problem hiding this comment.
It'll always be ordered and there are no plans to depart from five partitions. Else an ibterpolated structure or enum.Enum would work
There was a problem hiding this comment.
I would also favor an enum if we want these values exposed and consumable from other modules.
There was a problem hiding this comment.
ordering is important here, mutability isn't - happy to go with a tuple
garak/analyze/__init__.py
Outdated
| # SPDX-FileCopyrightText: Portions Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| SCORE_DEFCON_BOUNDS = [0.05, 0.4, 0.8, 0.99] |
There was a problem hiding this comment.
I would also favor an enum if we want these values exposed and consumable from other modules.
This PR allows a variety of group-level aggregations in reporting There are risks in using aggregated garak results, e.g. taking means of all probes in one category. Garak’s a discovery tool (not a benchmark) where anomalies are the signal - and some aggregation techniques, like averaging, are effective at eroding that signal. Two vignettes of how averaging makes garak results unusable: 1. Model A scores pretty well at all probes in a category. Model B scores the same but fails hard on one probe. Because there are many probes in the category, the mean shifts only a few percent, and the failure is completely missed. 2. A probe category has a high-risk and a low-risk probe. Model A scores 100% resilient at the high-risk one and 20% resilient at the low-risk one, and is approved for release. It gets a mean of 60%. Model B scores 100% resilient at the low-risk probe but 20% resilient at the high-risk probe, which is dangerous. However, the mean is still 60% like for model A, and no corrective action is flagged despite a high-risk weakness. The proposed change is to: * Add more aggregation options - e.g. minimum, median, lower quartile, mean minus standard deviation, proportion of failing detectors * Change the default aggregation technique used in HTML reports (tentatively will go with “minimum”, mean minus sd is cool too, so's lower quartile, @erickgalinkin wdyt?) This means (a) garak scores will drop, (b) improved visibility over model inference security. Additional changes: * report HTML has been cleared up, with descriptions moved to hover text, and duplicate content removed * `always.Random` detector that gives random scores in `0..1` - Tier-based changes are pending merge of #1151 - Hardcoded cutoff is present pending merge of #1144 - This continues to access `_config`; `report_digest` needs to be able to run standalone, and running it multithreaded is not intended to be supported ## Verification - Try to generate test results w/ e.g. `python -m garak -m test -p encoding,xss,ansiescape -d always.Random --report_prefix ~/dev/garak/test` (drop the use of `pxd` through `-d` to test Z-score changes) - Change the value in `_config.reporting.group_aggregation_function` through valid and unsupported ones, check that reports generate and look sane
* extract tier values from the plugin metadata * guard for divide by 0 Signed-off-by: Jeffrey Martin <jemartin@nvidia.com>
|
Do not merge until tier implementation is settled or held |
|
NB Currently #1152 should land /before/ this so that |
| TERRIBLE = -1.0 | ||
| BELOW_AVG = -0.125 | ||
| ABOVE_AVG = 0.125 | ||
| EXCELLENT = 1.0 |
There was a problem hiding this comment.
1 standard deviation here in a one-tailed test scenario would imply that an "excellent" score is higher than 90% of the distribution. Is that what is used across the domain? I am not familiar with these levels.
There was a problem hiding this comment.
Another comment: I stepped through the saw that the zscore value is derived from the scores distribution under the assumption that its Gaussian without checking for this (i.e. Shapiro-Wilk). Do we want to add such a check and warn over interpreting such values if normally cannot be confirmed? Or we could look at non-parametric tests for the scores distribution based on IQR or something similar?
There was a problem hiding this comment.
1 standard deviation here in a one-tailed test scenario would imply that an "excellent" score is higher than 90% of the distribution. Is that what is used across the domain? I am not familiar with these levels.
We set precendent here. We're grounding excellent in numbers. 86th percentil seems pretty good (isn't that where we are with a gaussian?)
Another comment: I stepped through the saw that the zscore value is derived from the scores distribution under the assumption that its Gaussian without checking for this (i.e. Shapiro-Wilk). Do we want to add such a check and warn over interpreting such values if normally cannot be confirmed? Or we could look at non-parametric tests for the scores distribution based on IQR or something similar?
Excellent spot. Shapiro-Wilk p-values scores are logged in calibration already; there's lots of bimodal stuff, I think. Plan is to change this to (a) keep track of all means to build a hierarchical model (b) give confidence/credible intervals of output scores based on that model
There was a problem hiding this comment.
86th percentil seems pretty good (isn't that where we are with a gaussian?)
Yes we get that. If we set precedent like you said then we can use this coding. Was just curious if this is something that has been used before.
Excellent spot. Shapiro-Wilk p-values scores are logged in calibration already; there's lots of bimodal stuff, I think. Plan is to change this to (a) keep track of all means to build a hierarchical model (b) give confidence/credible intervals of output scores based on that model
That's really interesting. Looking through that file it looks like the p-value is <0.01 for a lot of the metrics (under null hypothesis of normal distribution) probably as you said due to bimodality or skewed distributions. I like the idea of the hierarchical model.
Perhaps we can adapt the perf_stats module to add in other non-parametric measures of outliers like
There was a problem hiding this comment.
there will be a separate branch etc for this but yes, i would love to have this discussion!
There was a problem hiding this comment.
Sounds good, definitely some interesting work on calibrating scores' distributions' thresholds to detect "high" scores.
jmartin-tech
left a comment
jmartin-tech
left a comment
There was a problem hiding this comment.
Markdown is an improvement for output, we can iterate as this evolves.
3 participants
Standalone script that takes a report file as a CLI param, and performs standard analysis of probe/detector scores that are failing, taking into account tier 1 & tier 2 policies, and dumping out a sample of failing & passing inputs & outputs
Verification
python -m garak.analyze.qual_review garak.xxx.report.jsonl > xxx.qualitative.tsv