v0.14.0

What's Changed

New features

• feature: support JSON config as well as YAML by @patriciapampanelli in #1490
• feature: remove --generate_autodan CLI option by @sinhaabhiraj0 in #1521
• expand items in report entry_type:eval by @leondz in #1547
• reporting: tier biased security aggregate by @leondz in #1329
• feature: Report Generator refactor by @otavionvidia in #1573
• feature: Add detector evaluation doc and benchmark results by @patriciapampanelli in #1597
• task: calibration data updates winter 2026 by @jmartin-tech in #1599

Notice: Breaking changes in this release

• Modified format of eval and digest entires in JSONL report
  See "Run analysis" documentation for more details of compatibility expectations.
• Fully redesigned HTML reports
• Removal of --generate_autodan cli option

Improved plugins

• Fix atkgen verbose output displaying incorrect conversation turns by @snehalvartak in #1542
• generator: relax constraint to specify valid openai model name by @leondz in #1557
• generators: check message count requested matches message count generated by @leondz in #1558
• fix: JavaScript detector regex improvement by @erickgalinkin in #1588

Documentation

• docs: add guardion ai / promptintel to project by @leondz in #1530
• docs: add brief stubs for garak.analyze content by @leondz in #1569

Tuning & fixes

• Make GoogleTranslator multiprocessing-safe by @motlaharsh0909-lgtm in #1543
• Fix: improved unsafe pickle strategy by @jmartin-tech in #1545
• track which detector is reporting invalid pkg entry; only report once by @leondz in #1556
• arch: Remove many builtin configs by @leondz in #1555
• update: rm deprecated gpt-3.5 model from defaults by @leondz in #1552
• task: update openai library to 2.x by @jmartin-tech in #1550
• test: isolate _config in all tests by @jmartin-tech in #1549
• update LangChain generator for 1.x library support by @jmartin-tech in #1551
• add license to docs footer by @leondz in #1572
• guard for None content as response from target by @jmartin-tech in #1574
• fix: 100% pass rate should not give 'elevated risk' for relative scores by @leondz in #1579
• Move generation output cardinality check into Generator.generate() by @leondz in #1578
• calling as_dict() on default instantiated Attempt now doesn't raise TypeError by @leondz in #1590
• task: remove fschat by @erickgalinkin in #1567
• task: report test tweaks and dependency lock updates by @otavionvidia in #1593
• Task: transformers 5 support by @jmartin-tech in #1582
• task: exclude files not needed in distributed package by @jmartin-tech in #1601
• reporting: factor up defcon calculation by @leondz in #1581
• rm jinja2 direct dependency requirement by @leondz in #1596
• task: update huggingface config pattern by @jmartin-tech in #1602
• reporting: stabilise tbsa hashes by @leondz in #1603

New Contributors

• @snehalvartak made their first contribution in #1542
• @sinhaabhiraj0 made their first contribution in #1521
• @motlaharsh0909-lgtm made their first contribution in #1543
• @otavionvidia made their first contribution in #1573

Full Changelog: v0.13.3...v0.14.0

v0.13.3

What's Changed

New plugins

• probe: Api key probe + detector by @martinebl in #1406
• probe: zero width bad char injection by @Har1sh-k in #1489
• generator: AWS bedrock generator by @fabriziorocco in #1462

New features

• enable consistent PRNG access in generators by @jmartin-tech in #1496

Improved plugins

• probe: Add more unreal pkghallu targets by @leondz in #1480
• feature: OpenAICompatible classes do one generation at a time by default by @leondz in #1524

Documentation

• docs: add test for file extensions in docs dir by @leondz in #1474
• docs: describe probe module doc standard; implement it in all probe modules by @leondz in #1473
• docs(fix): Format list in RST by @mikemckiernan in #1491
• docs: add projects file for tracking garak ecosystem by @leondz in #1493
• doc: rm links to doc source; show links to github code source by @leondz in #1506
• docs: check for markdown in plugin docs, check for plugin module docstrings by @leondz in #1505
• docs: refine markdown checks by @leondz in #1510

Tuning & fixes

• tests: detector cardinality test tweaks by @leondz in #1476
• tool: add script for rebuilding plugin cache by @leondz in #1478
• allow for apikey case where group=None by @boltzmann-brain in #1497
• fix: metadata and vuln_id handling in AVID report export by @saichandrapandraju in #1495
• test: add tests for apikey detector None-group handling by @leondz in #1501
• bugfix: TreeSearchProbe should handle Nones by @leondz in #1509
• housekeeping: add code of conduct by @leondz in #1517
• arch: rm support for init'ing attempt.prompt with str, rm support for attempt.lang value by @leondz in #1361
• fix: encoding probes storing translated text in pre_translation_prompt by @paulinek13 in #1483
• feature: deferred loading and requirement pruning by @leondz in #1199

New Contributors

• @boltzmann-brain made their first contribution in #1497
• @fabriziorocco made their first contribution in #1462
• @saichandrapandraju made their first contribution in #1495
• @Har1sh-k made their first contribution in #1489

Full Changelog: v0.13.2...v0.13.3

v0.13.2

What's Changed

New plugins

• probe: Base class for iterative probes and FITD as an example by @aishwaryap in #1414
• detectors: any by @leondz in #1427

Improved plugins

• detectors: Fix incorrect target class for MustRefuteClaimModel by @RobGeada in #1419
• probes: Win11 probe still uses Windows 10 product names and goal by @2020ashish in #1422
• probes: properly parse the OFCOM list file by @paulinek13 in #1438
• probes: atkgen probe verbose output by @MrMoshkovitz in #1447
• probes: NameError in TAP/PAIR probes by @MrMoshkovitz in #1452
• detectors: access Message.text in PlainJSON by @jmartin-tech in #1454
• generators: enable parameter suppression to LiteLLM by @JDSmith-F1 in #1411

Documentation

• docs: revert unintended changes to translation configs by @paulinek13 in #1458
• docs: guide to running garak faster by @leondz in #1463
• docs: patch filename in doc on accelerating garak by @leondz in #1465

Tuning & fixes

• bugfix: transformers disconnect btw pegasus and transformers-community/group-beam-search by @leondz in #1403
• Use ABC to exclude abstract classes from --list_probes by @patriciapampanelli in #1398
• limit langchain version until migration by @jmartin-tech in #1420
• correct _mint_attempt conversation support by @jmartin-tech in #1417
• update github-specific content to 'target' param pattern by @leondz in #1426
• add init.py for all test paths containing .py by @jmartin-tech in #1428
• Fix/recommended detector migration by @patriciapampanelli in #1409
• refactor DAN probes using metaclass pattern by @patriciapampanelli in #1380
• update detectors to reflect immutable attempt prompt by @jmartin-tech in #1415
• improve prompt translation support for Conversation related types by @jmartin-tech in #1441
• fix: aggregate probes details in digest by @parkanzky in #1459
• fix: reduce window for multiprocessing race by @jmartin-tech in #1464

New Contributors

• @RobGeada made their first contribution in #1419
• @2020ashish made their first contribution in #1422
• @paulinek13 made their first contribution in #1438
• @MrMoshkovitz made their first contribution in #1447
• @JDSmith-F1 made their first contribution in #1411
• @parkanzky made their first contribution in #1459

Full Changelog: v0.13.1...v0.13.2

v0.13.1

What's Changed

New plugins

• probe: Atbash Encoding by @Nakul-Rajpal in #1343
• probe: ansi escape codes in tokenizer by @leondz in #1351
• dropbox repeated token attack by @dchiitmalla in #1244
• probe: DRA (Disguise and Reconstruction Attack) by @patriciapampanelli in #1345
• probes: package hallucination support for dart, perl, & raku by @dchiitmalla in #1243
• probe: Added token smuggling probe module by @mrowebot in #1192

New features

• feature: support detectors including None in output values by @leondz in #1280
• config: model_* to target_* by @leondz in #1383
• cli: add filtering support for --list_* options by @JosephDavisC in #1367

Improved plugins

• bugfix: future probes now use future phrasing by @leondz in #1388
• rename and extend Web injection probes by @erickgalinkin in #1335

Documentation

• fixing probable typo in cliref.rst by @cassiasamp in #1353
• docs(chore): Fix most build warnings by @mikemckiernan in #1359
• docs: reorganize table of concepts and getting started page by @mikemckiernan in #1360
• docs: theme recolor by @leondz in #1385
• docs: auto-include ASR in probe doc page if current calibration exists by @leondz in #1387
• bugfix: ASR figures in probe docs are now actual ASRs by @leondz in #1392

Tuning & fixes

• Specify flit build-system within pyproject.toml by @06kellyjac in #1317
• Update how license is defined in pyproject.toml by @06kellyjac in #1315
• reporting: add basic docs to scripts in analyze pkg by @leondz in #1369
• update: tektronix typo fix by @leondz in #1372
• cli: standardise reporting tool invocation patterns by @SubGlitch1 in #1370
• adding local fixture to setup wordnet environment by @dhruvmalik007 in #1342
• reporting: add aggregation of report jsonl digest objects, and tests for aggregation by @leondz in #1336
• update: rename data/misp_descriptions.tsv to data/tags.misp.tsv by @leondz in #1381

New Contributors

• @cassiasamp made their first contribution in #1353
• @Nakul-Rajpal made their first contribution in #1343
• @SubGlitch1 made their first contribution in #1370
• @dhruvmalik007 made their first contribution in #1342
• @patriciapampanelli made their first contribution in #1345
• @JosephDavisC made their first contribution in #1367

Full Changelog: v0.13.0...v0.13.1

v0.13.0

What's Changed

New plugins

• Doctor attack + encoding/Leet by @leondz in #1180
• Simple Assistive Task Linkage Probe by @erickgalinkin in #1319
• Ascii Smuggling by @erickgalinkin in #1299
• Added more generalized version of Markdown exfil probe by @aishwaryap in #1259

New features

• Feature: rename failure -> attack success by @leondz in #1326
• Feature: conversation support by @jmartin-tech in #1254
• Feature: Configurable system prompt by @erickgalinkin in #1337

Improved plugins

• Expand python exploitation payloads by @erickgalinkin in #1300
• Bug fix in python regex in malwaregen.AnyCode detector by @aishwaryap in #1296
• Bug fix to malwaregen anycode detector in keyword using by @aishwaryap in #1302
• Adding wider detection of :: and // in malwaregen.AnyCode detector by @aishwaryap in #1307
• Update datasets and regex for package hallucination by @arjun-krishna1 in #1124

Documentation

• amend docs copyright by @leondz in #1284
• documentation: typo fix by @dbaker-arch in #1286
• docs: improve conformance to PEP-0257 by @jmartin-tech in #1298
• docs: update contact email in readme by @leondz in #1341

Tuning & fixes

• Replace deprecated np.infty with np.inf. by @erickgalinkin in #1283
• restrict datasets version due to new torchcodec dependency by @jmartin-tech in #1290
• Promptinject cleanup by @erickgalinkin in #1292
• add missing import for re in azure generator by @jmartin-tech in #1294
• fix missing '.' in default config probe_spec by @jmartin-tech in #1295
• download models used in translation tests early by @jmartin-tech in #1308
• Tweaks to how wn is used by @06kellyjac in #1316
• Added NASM sections in regex for assembly in malwaregen.Anycode detector by @aishwaryap in #1310
• Use numpy v2 by @06kellyjac in #1314
• update usage of fschat required for compatibility by @jmartin-tech in #1322
• cohere version upgrade by @dchiitmalla in #1252
• improve cache and label automation by @jmartin-tech in #1324
• fix doc extending.rst by @solo-daemon in #1328
• reporting: add tags to digest, tidy naming by @leondz in #1334
• reporting: update defcon descriptions to talk about risk by @leondz in #1348
• judge detectors conversation refactor by @jmartin-tech in #1346

New Contributors

• @dbaker-arch made their first contribution in #1286
• @06kellyjac made their first contribution in #1316
• @solo-daemon made their first contribution in #1328

Full Changelog: v0.12.0...v0.13.0

v0.12.0

What's Changed

New plugins

• Add audio NIM model and audio probes by @erickgalinkin in #1163
• Leakreplay refactor by @dchiitmalla in #1264
• probes: refactor fact snippet mixin by @leondz in #1187

New features

• reporting: result summary object by @leondz in #1245
• rm octo generator by @leondz in #1248
• UX: add progress indicators for translation tasks by @jmartin-tech in #1257
• Feature: add google translate language provider by @jmartin-tech in #1232
• update LocalDataPath and testing for python 3.13 support by @jmartin-tech in #1188

Documentation

• docs: correct reference to langproviders configuration by @jmartin-tech in #1253
• chore: The bare .active has too much blast radius by @mikemckiernan in #1262
• docs: Address RST issues by @mikemckiernan in #1263
• chore: Update pre-commit hooks and use RST comment by @mikemckiernan in #1267
• docs: mistral syntax fix and minor code cleanup by @jmartin-tech in #1270
• chore: Reformat docstrings to reduce docs issues by @mikemckiernan in #1268

Tuning & fixes

• data correlation during calibration missed rename by @jmartin-tech in #1240
• bug fix: enable extended detectors, module-specified encoding payloads by default by @leondz in #1238
• adjust calling convention for perf_stats.py by @jmartin-tech in #1246
• remove modality redundant check by @jmartin-tech in #1251
• Llava tests by @dchiitmalla in #1256
• add action for closing stale PRs/issues by @leondz in #1269
• update refusal prompt by @katherine-luna in #1083
• add stale exempt tags by @leondz in #1272
• actions: add 'high priortiy' to never-stale label list by @leondz in #1274
• reset tiers for Complete leakreplay probes by @leondz in #1278

New Contributors

• @mikemckiernan made their first contribution in #1262
• @dchiitmalla made their first contribution in #1256
• @katherine-luna made their first contribution in #1083

Full Changelog: v0.11.0...v0.12.0

v0.11.0

What's Changed

New plugins

• Template / SQL injection probes by @erickgalinkin in #1138
• Feature/add mistral generator by @dimensi0n in #1135

New features

• feature: update default toxicity detector by @leondz in #1106
• feature: lightweight probe defaults by @leondz in #1116
• feature: max_workers / give kinda helpful message if too many open files by @leondz in #1110
• Multiprocess enabled logging config by @jmartin-tech in #1140
• Feature: multilingual machine translation by @SnowMasaya in #943
• Support stripping until end think token given empty skip_seq_start in config by @aishwaryap in #1185
• update: add probe tiers by @leondz in #1151
• update: promptinject detector now accepts multiple triggers by @leondz in #1148
• update: rename atkgen probe model to be clear about toxicity by @leondz in #1149
• update: remove ambiguous terms from slur_terms_en payload by @leondz in #1150
• reporting: update report aggregation funcs by @leondz in #1156
• script: qualitative review output by @leondz in #1144
• Add -no-cnv flag support to ggml generators by @IanYHChu in #1189
• reporting: add option for no group score by @leondz in #1194
• reporting: aggregate probe as min by @leondz in #1218
• reporting: add defcon lozenges for relative & absolute scores by @leondz in #1216
• Update/refactor specialwords by @leondz in #1178
• reporting: smooth z-score wildness by @leondz in #1212
• Task: 2025 Q2 scoring calibration by @jmartin-tech in #1231 (thanks to Vijil.ai for data contributions)
• update calibration data for additional probes by @jmartin-tech in #1236
• reporting: change default aggregation by @leondz in #1234

Documentation

• Fix typo in README for leak replay probe by @arjun-krishna1 in #1142
• docs: split 'extending' docs out from 'contributing' by @leondz in #1146
• doc file class corrections by @jmartin-tech in #1200
• docs: formatting fixes by @leondz in #1215

Tuning & fixes

• clear pip cached files by @jmartin-tech in #1129
• set a default soft_probe_prompt_cap in _config by @jmartin-tech in #1133
• enhance response type support from local NeMo-Guardrails by @jmartin-tech in #1131
• bugfix: encoding detection generating false positives by @leondz in #1130
• update: unify on attempt.notes["triggers"] by @leondz in #1147
• Bump datasets version by @JanetVictorious in #1137
• make all workflow permissions explicit by @jmartin-tech in #1162
• update: add soft prompt caps to encoding probes by @leondz in #1154
• update: rename bcp47 to lang by @leondz in #1164
• one detection result per output when testing regex based matches in exploitation by @jmartin-tech in #1167
• Removed detector prefix from eval records by @mrowebot in #1157
• bugfix: HF Detector exceptions now handled gracefully by default by @leondz in #1170
• cache workflow resources by @jmartin-tech in #1173
• refactor probe tier as enum with value in plugin cache by @jmartin-tech in #1159
• update: more meaningful values in tier enums by @leondz in #1176
• block failing litellm 1.67.2 by @leondz in #1179
• ux: give more verbose message for CLI typos by @leondz in #1182
• refactor LatentInjection by @leondz in #1152
• cap litellm max version to avoid their windows bug by @leondz in #1186
• update: rename Translator -> LangProvider and associated elements by @leondz in #1183
• bugfix: reduce latent optimisation permutation explosion by @leondz in #1181
• replicate generator pickle support improvements by @jmartin-tech in #1190
• Fix ambiguous series value error when running --report by @marcorosa in #1171
• add arm64 runner to Linux testing by @jmartin-tech in #1196
• Testing: storage reduction by @jmartin-tech in #1204
• remove unused tooling to free space by @jmartin-tech in #1206
• update deps away from insecure versions by @leondz in #1207
• update Tier impl by @leondz in #1205
• config: sync probe active defaults with default config used in practice by @leondz in #1214
• update: revert default _config.run.generations to 5 by @leondz in #1227
• fix: stop atkgen turn count variation in test relying on fixed turn count by @leondz in #1226
• fix plugin cache tests by @emmanuel-ferdman in #1229
• ux: move translator load msg into translator instantiation by @leondz in #1184
• extract text when processing multi-modal prompts by @jmartin-tech in #1228

New Contributors

• @JanetVictorious made their first contribution in #1137
• @SnowMasaya made their first contribution in #943
• @dimensi0n made their first contribution in #1135
• @mrowebot made their first contribution in #1157
• @aishwaryap made their first contribution in #1185
• @marcorosa made their first contribution in #1171
• @IanYHChu made their first contribution in #1189

Full Changelog: v0.10.3.1...v0.11.0

v0.10.3.1

What's Changed

Tuning & fixes

• defensive coding around capture results in xss content detector by @jmartin-tech #1120

Full Changelog: v0.10.3...v0.10.3.1

v0.10.3

What's Changed

New plugins

• Added watsonx.ai generator by @iamnotcj in #1058
• Additional XSS Exfil Probes by @erickgalinkin in #1060

New features

• Generators: add option to skip output btw given delimiters; add hook for postprocessing by @leondz in #1097
• Feature: align hallucinated package named with outputs by @leondz in #1076
• support module.classname config specification for plugins by @jmartin-tech in #1108

Documentation

• pedantic spelling by @dltemple in #1085
• update contributing checklist numbers to be ascending by @shane-rosse in #1115

Tuning & fixes

• update: reorder ditw prompts by @leondz in #1098
• update openai model list by @leondz in #1100
• update: Give clearer message when header encoding fails by @leondz in #1088

What's Changed

• uninclude donotanswer from default probes by @leondz in #1111
• stop forcing generation count for mini phrasing probes by @leondz in #1109

New Contributors

• @iamnotcj made their first contribution in #1058
• @dltemple made their first contribution in #1085
• @shane-rosse made their first contribution in #1115

Full Changelog: v0.10.2...v.0.10.3

v0.10.2

What's Changed

New plugins

• Detector: Shields for testing LLM Application Firewalls by @Eric-Hacker in #1059

New features

• Warn if api_key in Config by @erickgalinkin in #1049
• Feature: configuration based rest proxy support by @jmartin-tech in #1073
• Feature: configuration based rest ssl suppression by @jmartin-tech in #1074
• Detector: add startswith matching for StringDetector base class by @leondz in #1075

Documentation

• docs: update index.rst by @eltociear in #1053
• docs: update readme by @leondz in #1050
• adjust label ref syntax by @jmartin-tech in #1057

Tuning & fixes

• StringDetector.case_sensitive now operates as expected by @leondz in #1072
• guard against None in leakreplay attempt history management by @leondz in #1081
• init zscore with enough scope to cover its uses by @leondz in #1086
• Update for latest ollama support by @jmartin-tech in #1092

New Contributors

• @eltociear made their first contribution in #1053
• @Eric-Hacker made their first contribution in #1059

Full Changelog: v0.10.1...v0.10.2