v0.11.0 · NVIDIA garak · Discussion #1237 · GitHub

jmartin-tech
May 28, 2025
Maintainer

What's Changed

New plugins

Template / SQL injection probes by @erickgalinkin in Template / SQL injection probes #1138
Feature/add mistral generator by @dimensi0n in Feature/add mistral generator #1135

New features

feature: update default toxicity detector by @leondz in feature: update default toxicity detector #1106
feature: lightweight probe defaults by @leondz in feature: lightweight probe defaults #1116
feature: max_workers / give kinda helpful message if too many open files by @leondz in feature: max_workers / give kinda helpful message if too many open files #1110
Multiprocess enabled logging config by @jmartin-tech in Multiprocess enabled logging config #1140
Feature: multilingual machine translation by @SnowMasaya in Feature: multilingual machine translation #943
Support stripping until end think token given empty skip_seq_start in config by @aishwaryap in Support stripping until end think token given empty skip_seq_start in config #1185
update: add probe tiers by @leondz in update: add probe tiers #1151
update: promptinject detector now accepts multiple triggers by @leondz in update: promptinject detector now accepts multiple triggers #1148
update: rename atkgen probe model to be clear about toxicity by @leondz in update: rename atkgen probe model to be clear about toxicity #1149
update: remove ambiguous terms from slur_terms_en payload by @leondz in update: remove ambiguous terms from slur_terms_en payload #1150
reporting: update report aggregation funcs by @leondz in reporting: update report aggregation funcs #1156
script: qualitative review output by @leondz in script: qualitative review output #1144
Add -no-cnv flag support to ggml generators by @IanYHChu in Add -no-cnv flag support to ggml generators #1189
reporting: add option for no group score by @leondz in reporting: add option for no group score #1194
reporting: aggregate probe as min by @leondz in reporting: aggregate probe as min #1218
reporting: add defcon lozenges for relative & absolute scores by @leondz in reporting: add defcon lozenges for relative & absolute scores #1216
Update/refactor specialwords by @leondz in Update/refactor specialwords #1178
reporting: smooth z-score wildness by @leondz in reporting: smooth z-score wildness #1212
Task: 2025 Q2 scoring calibration by @jmartin-tech in Task: 2025 Q2 scoring calibration #1231 (thanks to Vijil.ai for data contributions)
update calibration data for additional probes by @jmartin-tech in update calibration data for additional probes #1236
reporting: change default aggregation by @leondz in reporting: change default aggregation #1234

Documentation

Fix typo in README for leak replay probe by @arjun-krishna1 in Fix typo in README for leak replay probe #1142
docs: split 'extending' docs out from 'contributing' by @leondz in docs: split 'extending' docs out from 'contributing' #1146
doc file class corrections by @jmartin-tech in doc file class corrections #1200
docs: formatting fixes by @leondz in docs: formatting fixes #1215

Tuning & fixes

clear pip cached files by @jmartin-tech in clear pip cached files #1129
set a default soft_probe_prompt_cap in _config by @jmartin-tech in set a default soft_probe_prompt_cap in _config #1133
enhance response type support from local NeMo-Guardrails by @jmartin-tech in enhance response type support from local NeMo-Guardrails #1131
bugfix: encoding detection generating false positives by @leondz in bugfix: encoding detection generating false positives #1130
update: unify on attempt.notes["triggers"] by @leondz in update: unify on attempt.notes["triggers"] #1147
Bump datasets version by @JanetVictorious in Bump datasets version #1137
make all workflow permissions explicit by @jmartin-tech in make all workflow permissions explicit #1162
update: add soft prompt caps to encoding probes by @leondz in update: add soft prompt caps to encoding probes #1154
update: rename bcp47 to lang by @leondz in update: rename bcp47 to lang #1164
one detection result per output when testing regex based matches in exploitation by @jmartin-tech in one detection result per output when testing regex based matches in exploitation #1167
Removed detector prefix from eval records by @mrowebot in Removed detector prefix from eval records #1157
bugfix: HF Detector exceptions now handled gracefully by default by @leondz in bugfix: HF Detector exceptions now handled gracefully by default #1170
cache workflow resources by @jmartin-tech in cache workflow resources #1173
refactor probe tier as enum with value in plugin cache by @jmartin-tech in refactor probe tier as enum with value in plugin cache #1159
update: more meaningful values in tier enums by @leondz in update: more meaningful values in tier enums #1176
block failing litellm 1.67.2 by @leondz in block failing litellm 1.67.2 #1179
ux: give more verbose message for CLI typos by @leondz in ux: give more verbose message for CLI typos #1182
refactor LatentInjection by @leondz in refactor LatentInjection #1152
cap litellm max version to avoid their windows bug by @leondz in cap litellm max version to avoid their windows bug #1186
update: rename Translator -> LangProvider and associated elements by @leondz in update: rename Translator -> LangProvider and associated elements #1183
bugfix: reduce latent optimisation permutation explosion by @leondz in bugfix: reduce latent optimisation permutation explosion #1181
replicate generator pickle support improvements by @jmartin-tech in replicate generator pickle support improvements #1190
Fix ambiguous series value error when running --report by @marcorosa in Fix ambiguous series value error when running --report #1171
add arm64 runner to Linux testing by @jmartin-tech in add arm64 runner to Linux testing #1196
Testing: storage reduction by @jmartin-tech in Testing: storage reduction #1204
remove unused tooling to free space by @jmartin-tech in remove unused tooling to free space #1206
update deps away from insecure versions by @leondz in update deps away from insecure versions #1207
update Tier impl by @leondz in update Tier impl #1205
config: sync probe active defaults with default config used in practice by @leondz in config: sync probe active defaults with default config used in practice #1214
update: revert default _config.run.generations to 5 by @leondz in update: revert default _config.run.generations to 5 #1227
fix: stop atkgen turn count variation in test relying on fixed turn count by @leondz in fix: stop atkgen turn count variation in test relying on fixed turn count #1226
fix plugin cache tests by @emmanuel-ferdman in fix plugin cache tests #1229
ux: move translator load msg into translator instantiation by @leondz in ux: move translator load msg into translator instantiation #1184
extract text when processing multi-modal prompts by @jmartin-tech in extract text when processing multi-modal prompts #1228

New Contributors

@JanetVictorious made their first contribution in Bump datasets version #1137
@SnowMasaya made their first contribution in Feature: multilingual machine translation #943
@dimensi0n made their first contribution in Feature/add mistral generator #1135
@mrowebot made their first contribution in Removed detector prefix from eval records #1157
@aishwaryap made their first contribution in Support stripping until end think token given empty skip_seq_start in config #1185
@marcorosa made their first contribution in Fix ambiguous series value error when running --report #1171
@IanYHChu made their first contribution in Add -no-cnv flag support to ggml generators #1189

Full Changelog: v0.10.3.1...v0.11.0

This discussion was created from the release v0.11.0.

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment