Support for openai v2.x API

[saichandrapandraju]

Hi team!

I'm encountering a dependency conflict when trying to use garak in an environment with openai v2.x API (used by other key packages).

1. Is there a known blocker preventing garak from supporting openai v2.x?
2. Are there plans/timeline to upgrade the openai dependency upper bound?

Happy to help test or contribute if needed. Thanks!

[leondz]

I think we'd like to support this, are you able to give some concrete motivating examples of what using the earlier version blocks?

[saichandrapandraju]

Sure. We run garak scans using llama-stack via KFP (as an external eval provider). The recent 0.4.x versions of llama-stack use openai>=2.5.0 and this is blocking from using garak scans with latest distributions.

Since there's no common openai package, our image builds fail -

  × No solution found when resolving dependencies:
  ╰─▶ Because llama-stack==0.4.0 depends on openai>=2.5.0 and garak==0.12.0
      depends on openai>=1.45.0,<2, we can conclude that garak==0.12.0 and
      llama-stack[client]==0.4.0 are incompatible.
      And because llama-stack-provider-trustyai-garak depends on
      llama-stack[client]==0.4.0 and garak==0.12.0, we can conclude that your
      requirements are unsatisfiable.

[leondz]

Thanks. Agree this doesn't work, we'll take a look.