Add Bootstrap Confidence Intervals for Attack Success Rates#1577

Open

patriciapampanelli wants to merge 26 commits intoNVIDIA:mainfrom

patriciapampanelli:feature/confidence-intervals

Commits on Feb 6, 2026

Implement non-parametric bootstrap CI for ASR with detector correction
patriciapampanelli
committed
Display CI's as asymmetric bounds [lower%, upper%] instead of symmetric ± format
patriciapampanelli
committed
Update garak/analyze/bootstrap_ci.py

patriciapampanelli
and
erickgalinkin
committed
Optimize calculation by skipping correction for perfect detectors
patriciapampanelli
committed
Update docs/source/reporting.rst

patriciapampanelli
and
leondz
committed
Centralize configuration values and improve error handling
patriciapampanelli
committed
Refactor detector metrics exception handling
patriciapampanelli
committed
Refactor detector metrics and document bootstrap config
patriciapampanelli
committed
Make CI calculation optional via reporting.confidence_interval_method config with None/bootstrap options
patriciapampanelli
committed
Add --rebuild_cis command and fix CI inversion bug in HTML display
patriciapampanelli
committed
Add CI calculator tests
patriciapampanelli
committed
Hide zero-width bootstrap CIs at boundaries (0% and 100%)
patriciapampanelli
committed
clarify confidence interval configuration in reporting
patriciapampanelli
committed
standardize confidence_interval_method config to use empty values
patriciapampanelli
committed
Add type hints for optional CI parameters in evaluators
patriciapampanelli
committed
Update garak/cli.py

patriciapampanelli
and
leondz
committed
Simplify bootstrap data collection using post-hoc construction
patriciapampanelli
committed
Use config default for eval_threshold in ci_calculator fallback
patriciapampanelli
committed
Move bootstrap_min_sample_size to config for single source of truth
patriciapampanelli
committed
Refactor _get_detectors_info to return dicts instead of tuples
patriciapampanelli
committed
Refactor CI calculation to use digest aggregates instead of individual attempts
patriciapampanelli
committed
Fix merge conflict artifacts in evaluators/base.py
patriciapampanelli
committed
missing comma in SQL CREATE TABLE
patriciapampanelli
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Bootstrap Confidence Intervals for Attack Success Rates#1577

Add Bootstrap Confidence Intervals for Attack Success Rates#1577
patriciapampanelli wants to merge 26 commits intoNVIDIA:mainfrom
patriciapampanelli:feature/confidence-intervals

Commits on Feb 6, 2026

Implement non-parametric bootstrap CI for ASR with detector correction

Display CI's as asymmetric bounds [lower%, upper%] instead of symmetric ± format

Update garak/analyze/bootstrap_ci.py

Optimize calculation by skipping correction for perfect detectors

Update docs/source/reporting.rst

Centralize configuration values and improve error handling

Refactor detector metrics exception handling

Refactor detector metrics and document bootstrap config

Make CI calculation optional via reporting.confidence_interval_method config with None/bootstrap options

Add --rebuild_cis command and fix CI inversion bug in HTML display

Add CI calculator tests

Hide zero-width bootstrap CIs at boundaries (0% and 100%)

clarify confidence interval configuration in reporting

standardize confidence_interval_method config to use empty values

Add type hints for optional CI parameters in evaluators

Update garak/cli.py

Simplify bootstrap data collection using post-hoc construction

Use config default for eval_threshold in ci_calculator fallback

Move bootstrap_min_sample_size to config for single source of truth

Refactor _get_detectors_info to return dicts instead of tuples

Refactor CI calculation to use digest aggregates instead of individual attempts

Fix merge conflict artifacts in evaluators/base.py

missing comma in SQL CREATE TABLE

Commits on Feb 19, 2026

Update garak/evaluators/base.py

Update garak/evaluators/base.py

Store bootstrap confidence level as float in eval record