Skip to content

fix(openai): infer model input modalities to prevent image read failures#2365

Merged
amitksingh1490 merged 2 commits intomainfrom
open-ai-models-modality
Feb 8, 2026
Merged

fix(openai): infer model input modalities to prevent image read failures#2365
amitksingh1490 merged 2 commits intomainfrom
open-ai-models-modality

Conversation

@amitksingh1490
Copy link
Copy Markdown
Contributor

Summary

Fix OpenAI model modality discovery by inlining a curated model catalog (including input_modalities) so vision-capable models can be selected reliably and image reading no longer fails.
This avoids depending on the OpenAI Models API for modality metadata, which is currently missing/incomplete.

Context

The OpenAI Models API does not consistently return modality information, which breaks Forge’s ability to detect image-capable models and causes image read flows to fail.

Changes

  • Replaced the OpenAI provider models endpoint URL with an inline model catalog that includes modality metadata.
  • Added input_modalities entries (e.g. text, image) to ensure the UI/selection logic can reliably choose vision-capable models.

Key Implementation Details

  • The OpenAI provider entry in provider.json now contains an explicit models array rather than a remote URL.
  • Each model includes context_length, tool-call capabilities, and input_modalities to drive capability-aware selection.

Use Cases

  • Selecting a model for image reading now works because vision-capable models are explicitly marked with input_modalities: ["text", "image"].
  • Local/offline workflows no longer require an additional API call to enumerate models.

Testing

cargo test -p forge_repo

Notes

  • The model list is curated; it can be updated as OpenAI adds/changes models or as modality metadata becomes reliably available via the API.

@github-actions github-actions Bot added the type: fix Iterations on existing features or infrastructure. label Feb 8, 2026
@amitksingh1490 amitksingh1490 merged commit 32523b9 into main Feb 8, 2026
11 checks passed
@amitksingh1490 amitksingh1490 deleted the open-ai-models-modality branch February 8, 2026 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: fix Iterations on existing features or infrastructure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant