fix: metadata and vuln_id handling in AVID report export

[saichandrapandraju]

Fix AVID report export errors related to metadata keys and the default vuln_id. details in this comment.

Changes:

• Updated entry type check from config to start_run setup.
• Updated metadata references to correctly access plugin attributes for target name and type.
• Added a default value (None) for vuln_id in the impact section.

Fixes:

• Resolves TypeError: 'NoneType' object is not subscriptable when metadata is accessed
• Resolves KeyError: 'target_type' when accessing metadata
• Resolves pydantic validation error: "Field required [type=missing]" for vuln_id

Verification

List the steps needed to make sure this thing works

• Run garak -r <report_path> and confirm that AVID reports are generated successfully.

[leondz]

Thanks for this! Do you have a pointer to reproducing the fixed errors? They sound like they'd make decent tests

[leondz]

The code looks good! Would appreciate info on test cases

[erickgalinkin]

+1 to Leon's comments. Would love to see the failing test cases but otherwise lgtm.

[saichandrapandraju]

Hi @leondz & @erickgalinkin, thank you for looking into this.

For the below sample run -

> garak --probes dan.AntiDAN -t openai.OpenAICompatible -n qwen2 --generator_options '{"openai": {"OpenAICompatible": {"uri": "http://localhost:8080/v1", "model": "qwen2", "api_key": "DUMMY", "suppressed_params": ["n"]}}}'
garak LLM vulnerability scanner v0.13.3.pre1 ( https://github.com/NVIDIA/garak ) at 2025-11-21T09:31:58.145498
📜 logging to /Users/spandraj/.local/share/garak/garak.log
🦜 loading generator: OpenAICompatible: qwen2
📜 reporting to /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.jsonl
🕵️  queue of probes: dan.AntiDAN
dan.AntiDAN                                                                              dan.AntiDAN: FAIL  ok on    0/   1   (attack success rate: 100.00%)              
dan.AntiDAN                                                              mitigation.MitigationBypass: PASS  ok on    1/   1
📜 report closed :) /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.jsonl
📜 report html summary being written to /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.html
✔️  garak run complete in 4.75s

Before any fixes:

garak -r /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.jsonl
garak LLM vulnerability scanner v0.13.3.pre1 ( https://github.com/NVIDIA/garak ) at 2025-11-21T09:32:49.338839
/Users/spandraj/.pyenv/versions/3.12.9/envs/garak-dev/lib/python3.12/site-packages/pydantic/_internal/_config.py:383: UserWarning: Valid config keys have changed in V2:
* 'fields' has been removed
  warnings.warn(message, UserWarning)
📜 Converting garak reports /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.jsonl
Traceback (most recent call last):
  File "/Users/spandraj/.pyenv/versions/garak-dev/bin/garak", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/spandraj/dev/redteam_playground/garak/garak/__main__.py", line 9, in main
    cli.main(sys.argv[1:])
  File "/Users/spandraj/dev/redteam_playground/garak/garak/cli.py", line 541, in main
    report.export()
  File "/Users/spandraj/dev/redteam_playground/garak/garak/report.py", line 121, in export
    value=f"The model {self.metadata['plugins.target_name']} from {self.metadata['plugins.target_type']} was evaluated by the Garak LLM Vunerability scanner using the probe `{probe}`.",
                       ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable

After the 1st update (entry type check from config to start_run setup)

> garak -r /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.jsonl
garak LLM vulnerability scanner v0.13.3.pre1 ( https://github.com/NVIDIA/garak ) at 2025-11-21T09:33:29.915263
/Users/spandraj/.pyenv/versions/3.12.9/envs/garak-dev/lib/python3.12/site-packages/pydantic/_internal/_config.py:383: UserWarning: Valid config keys have changed in V2:
* 'fields' has been removed
  warnings.warn(message, UserWarning)
📜 Converting garak reports /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.jsonl
Traceback (most recent call last):
  File "/Users/spandraj/.pyenv/versions/garak-dev/bin/garak", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/spandraj/dev/redteam_playground/garak/garak/__main__.py", line 9, in main
    cli.main(sys.argv[1:])
  File "/Users/spandraj/dev/redteam_playground/garak/garak/cli.py", line 541, in main
    report.export()
  File "/Users/spandraj/dev/redteam_playground/garak/garak/report.py", line 95, in export
    deployer=[self.metadata["target_type"]],
              ~~~~~~~~~~~~~^^^^^^^^^^^^^^^
KeyError: 'target_type'

After the 2nd update (metadata references to correctly access plugin.* attributes for target name and type.)

> garak -r /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.jsonl
garak LLM vulnerability scanner v0.13.3.pre1 ( https://github.com/NVIDIA/garak ) at 2025-11-21T09:33:59.326421
/Users/spandraj/.pyenv/versions/3.12.9/envs/garak-dev/lib/python3.12/site-packages/pydantic/_internal/_config.py:383: UserWarning: Valid config keys have changed in V2:
* 'fields' has been removed
  warnings.warn(message, UserWarning)
📜 Converting garak reports /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.jsonl
1 validation error for AvidTaxonomy
vuln_id
  Field required [type=missing, input_value={'risk_domain': ['Securit... 'taxonomy_version': ''}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing

After the final 3rd update (default value (None) for vuln_id in the impact section.)

> garak -r /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.jsonl
garak LLM vulnerability scanner v0.13.3.pre1 ( https://github.com/NVIDIA/garak ) at 2025-11-21T09:34:08.855430
/Users/spandraj/.pyenv/versions/3.12.9/envs/garak-dev/lib/python3.12/site-packages/pydantic/_internal/_config.py:383: UserWarning: Valid config keys have changed in V2:
* 'fields' has been removed
  warnings.warn(message, UserWarning)
📜 Converting garak reports /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.report.jsonl
📜 AVID reports generated at /Users/spandraj/.local/share/garak/garak_runs/garak.0d763de6-1cd6-46b0-9d50-51a546abd672.avid.jsonl

They sound like they'd make decent tests

Absolutely, I can create a test file for this report.py. I might have to overwrite/replace tests/_assests/test.report.jsonl with a similar report generated with latest garak to use target_type and target_name in order to use it for potential tests/test_report.py, is it okay to do so?

[leondz]

Thanks!

I might have to overwrite/replace tests/_assests/test.report.jsonl with a similar report generated with latest garak to use target_type and target_name in order to use it for potential tests/test_report.py, is it okay to do so?

Is it possible to create a new file in _assets, e.g. report_test.report.jsonl?

[saichandrapandraju]

Ofc, thanks for the clarification. Will create a tests/test_report.py in a bit.

[saichandrapandraju]

Curious why name is an empty string.

[leondz]

@shubhobm Can you help here?

[jmartin-tech]

Changes here have been validated for resolution of the reported failure. This will land shortly.

The question about naming the reported metric is still open to the original feature author.

Requested testing has been supplied.