Capability audit for the 13 text/multimodal AI providers historically tracked in this matrix. NeuroLink ships 21+ providers in total — the additional providers added since this audit was first written (DeepSeek, NVIDIA NIM, LM Studio, llama.cpp) and the voice providers (OpenAI TTS, ElevenLabs, Deepgram, Azure Speech, Whisper, OpenAI Realtime, Gemini Live) are documented in the per-provider docs under /docs/providers/ and the Voice Features index, not in this capability matrix.
For the canonical product surface, see the README.
Last Updated: May 2026 NeuroLink Version: 9.62.0
| Provider | Text Gen | Streaming | Tools | Vision | Thinking | Structured Output | Auth Required | |
|---|---|---|---|---|---|---|---|---|
| OpenAI | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | API Key |
| Anthropic | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | API Key |
| Google AI Studio | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | API Key | |
| Google Vertex | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Service Account | |
| Amazon Bedrock | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | AWS Credentials | |
| Amazon SageMaker | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | AWS Credentials | |
| Azure OpenAI | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | API Key + Endpoint |
| Mistral | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | API Key | |
| HuggingFace | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | API Key | |
| LiteLLM | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | Custom | |
| Ollama | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | None | |
| OpenAI Compatible | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | Custom | |
| OpenRouter | ✓ | ✓ | ✗ | ✗ | ✓ | API Key |
Legend:
- ✓ Full Support
⚠️ Partial/Model-Dependent Support- ✗ Not Supported
File: src/lib/providers/openAI.ts
Provider Name: openai
Default Model: gpt-4o
- Full support for all GPT models
- Supports temperature, maxTokens, top_p parameters
- Multi-turn conversations
- Real-time token streaming via Server-Sent Events (SSE)
- Chunk-by-chunk response delivery
- Full analytics support
- Native function calling support
- Automatic tool execution
- Multi-step tool workflows
- Tool choice: auto, required, none
Supported Models:
- GPT-5.2 series (gpt-5.2, gpt-5.2-pro) - Latest flagship
- GPT-5 series (gpt-5, gpt-5-pro, gpt-5-mini, gpt-5-nano)
- GPT-4.1 series (gpt-4.1, gpt-4.1-mini, gpt-4.1-nano)
- O-series reasoning models (o3, o3-mini, o3-pro, o4, o4-mini)
- GPT-4o, GPT-4o-mini, GPT-4-turbo, GPT-4-vision-preview
Image Support:
- Up to 10 images per request
- Formats: PNG, JPEG, WEBP, GIF
- Base64 and URL input
- Not natively supported
- Requires external preprocessing
- Standard reasoning only
- No extended thinking capability
- JSON schema validation
- Type-safe responses via Zod
- Response format enforcement
# Required
OPENAI_API_KEY=sk-...
# Optional
OPENAI_MODEL=gpt-4o
OPENAI_BASE_URL=https://api.openai.com/v1 # For proxy/custom endpoints- PDF files require preprocessing to text/images
- No native extended thinking mode
- Rate limits apply per API key tier
- Context window varies by model (128K for GPT-4o)
File: src/lib/providers/anthropic.ts
Provider Name: anthropic
Default Model: claude-sonnet-4-5-20250929
- All Claude models (3.x, 4.x, 4.5)
- Advanced reasoning capabilities
- Long context support (200K tokens)
- Real-time streaming with SSE
- Tool execution during streaming
- Analytics tracking
- Native tool use support
- Multi-step agentic workflows
- Tool result caching
- Parallel tool execution
Supported Models:
- Claude 4.5 series (Sonnet, Opus, Haiku)
- Claude 4.1 and 4.0 series
- Claude 3.7 series
- Claude 3.5 series
- Claude 3 series (Opus, Sonnet, Haiku)
Image Support:
- Up to 20 images per request
- Formats: PNG, JPEG, WEBP, GIF
- Base64 encoding required
- Native PDF document understanding
- No preprocessing required
- Extract text, tables, and structure
- Visual analysis of PDF pages
Supported Models:
- Claude 4.5 Sonnet (latest)
- Claude 4.5 Opus
- Claude 4.1 Opus
- Claude 3.7 Sonnet
Thinking Levels:
minimal- Fast responseslow- Basic reasoningmedium- Moderate reasoning (default)high- Deep reasoning and analysis
- JSON schema validation
- Type-safe responses
- Zod schema support
# Required
ANTHROPIC_API_KEY=sk-ant-...
# Optional
ANTHROPIC_MODEL=claude-sonnet-4-5-20250929
ANTHROPIC_VERSION=2023-06-01- 200K token context window (generous but finite)
- API rate limits based on tier
- Extended thinking increases latency
- PDF processing has file size limits
File: src/lib/providers/googleAiStudio.ts
Provider Name: google-ai / googleAiStudio
Default Model: gemini-2.5-flash
- Gemini 1.5, 2.0, 2.5, and 3.0 models
- Fast inference
- Free tier available
- Real-time streaming
- Tool execution during streaming
- Analytics support
- Native function calling
- Parallel tool execution
- Tool result integration
Supported Models:
- Gemini 3 series (Pro, Flash) - Preview
- Gemini 2.5 series (Pro, Flash, Flash Lite)
- Gemini 2.0 series (Flash)
- Gemini 1.5 series (Pro, Flash)
Image Support:
- Up to 16 images per request
- Formats: PNG, JPEG, WEBP
- Base64 and Google Cloud Storage URLs
- Native PDF understanding
- Text and visual extraction
- Document structure analysis
Supported Models:
- Gemini 3 Pro (Preview)
- Gemini 2.5 Pro
- Gemini 2.5 Flash
Thinking Levels:
minimal,low,medium,high- Configurable thinking budget
- JSON schema support
- CRITICAL LIMITATION: Cannot use tools AND structured output simultaneously
- When using JSON schema, must set
disableTools: true - Error: "Function calling with response mime type 'application/json' is unsupported"
# Required
GOOGLE_AI_API_KEY=AIza...
# Optional
GOOGLE_AI_MODEL=gemini-2.5-flash- Cannot combine tools + JSON schema (Gemini limitation)
- Tools OR structured output, not both
- Free tier has rate limits
- Some features in preview/experimental
File: src/lib/providers/googleVertex.ts
Provider Name: vertex
Default Model: gemini-2.5-flash
Same as Google AI Studio, plus:
- Gemini models - Same as AI Studio
- Claude models via Vertex - Anthropic models hosted on GCP
Anthropic on Vertex:
- Claude 4.5 series (Sonnet, Opus, Haiku)
- Claude 4.x and 3.x series
- Full tool calling support
- No structured output limitation (unlike Gemini)
- All Gemini models
- All Claude models via Vertex Anthropic
- Enterprise-grade reliability
- Same as AI Studio
- Works for both Gemini and Claude models
- Gemini: Full tool support (but not with schemas)
- Claude: Full tool support (can combine with schemas)
- Gemini: Up to 16 images
- Claude: Up to 20 images
- Both Gemini and Claude models support PDF
- Gemini 2.5+, Gemini 3: Full support
- Claude models: Not supported via Vertex
- Gemini: Cannot combine with tools
- Claude: Can combine with tools
# Required (Option 1: Service Account File)
GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
VERTEX_PROJECT_ID=my-project
# Required (Option 2: Environment Variables)
GOOGLE_AUTH_CLIENT_EMAIL=...
GOOGLE_AUTH_PRIVATE_KEY=...
VERTEX_PROJECT_ID=my-project
# Optional
VERTEX_LOCATION=us-central1
VERTEX_MODEL=gemini-2.5-flash- Requires Google Cloud project setup
- Service account authentication complexity
- Gemini tools + schema limitation applies
- Regional endpoint configuration
File: src/lib/providers/amazonBedrock.ts
Provider Name: bedrock
Default Model: anthropic.claude-3-sonnet-20240229-v1:0
- Claude models on Bedrock
- Amazon Titan models
- Cohere models
- Meta Llama models
- AI21 Jurassic models
- Real-time streaming via AWS SDK
- Native conversation loop
- Tool execution during streaming
- Native tool support via Bedrock Converse API
- Multi-step tool workflows
- Automatic tool execution
Model-Dependent:
- Claude models: Full vision support
- Titan models: Limited vision support
- Other models: Varies by model
- Claude models: Native PDF support
- Document extraction and analysis
- Not supported via Bedrock
- Standard reasoning only
- JSON schema validation
- Type-safe responses
# Required
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1
# Optional
BEDROCK_MODEL=anthropic.claude-3-sonnet-20240229-v1:0- Requires AWS account with Bedrock access
- Model availability varies by region
- IAM permissions required
- No extended thinking support
- Vision support depends on model
File: src/lib/providers/amazonSagemaker.ts
Provider Name: sagemaker
Default Model: Custom endpoint
- Custom SageMaker endpoints
- Fine-tuned models
- Enterprise model deployments
- Not fully implemented for SageMaker custom endpoints. Streaming returns a 501 error from SageMaker custom inference endpoints; non-streaming generation works.
- Supported for compatible models
- Depends on endpoint configuration
- Not supported
- Depends on custom endpoint
- Not supported
- Not supported
- Not supported via provider
- May work with custom endpoints
# Required
AWS_ACCESS_KEY_ID=AKIA...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1
SAGEMAKER_ENDPOINT_NAME=my-endpoint
# Optional
SAGEMAKER_MODEL=custom-model- Streaming not fully implemented
- Requires SageMaker endpoint deployment
- Custom model-dependent capabilities
- No built-in multimodal support
- Enterprise AWS setup required
File: src/lib/providers/azureOpenai.ts
Provider Name: azure
Default Model: gpt-4o
- All Azure OpenAI models
- GPT-4, GPT-4o, GPT-3.5-turbo
- Enterprise security and compliance
- Real-time streaming
- Tool execution during streaming
- Analytics support
- Full tool support
- Same as OpenAI provider
- Multi-step workflows
Supported Models:
- GPT-5.1 series
- GPT-5 series
- GPT-4.1 series
- O-series (o3, o4)
- GPT-4o, GPT-4o-mini, GPT-4-turbo
Image Support:
- Up to 10 images per request
- Same formats as OpenAI
- Not natively supported
- Not supported
- JSON schema validation
- Type-safe responses
# Required
AZURE_OPENAI_API_KEY=...
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
AZURE_OPENAI_DEPLOYMENT=gpt-4o
# Optional
AZURE_API_VERSION=2024-05-01-preview- Requires Azure subscription
- Deployment configuration required
- Regional model availability varies
- No PDF or extended thinking support
File: src/lib/providers/mistral.ts
Provider Name: mistral
Default Model: mistral-small-2506
- Mistral Small, Medium, Large models
- Fast inference
- Cost-effective
- Real-time streaming
- Tool execution support
- Native function calling
- Tool execution workflows
Supported Models:
- Mistral Small 2506 (June 2025) - Vision-capable
- Mistral Pixtral - Multimodal model
Image Support:
- Up to 10 images per request (conservative limit)
- Model-dependent capability
- Not supported
- Not supported
- JSON schema support
- Type-safe responses
# Required
MISTRAL_API_KEY=...
# Optional
MISTRAL_MODEL=mistral-small-2506- Vision only on specific models (Small 2506+)
- No PDF support
- No extended thinking
- Limited multimodal compared to GPT-4o/Claude
File: src/lib/providers/huggingFace.ts
Provider Name: huggingface
Default Model: microsoft/DialoGPT-medium
- Access to 100,000+ models
- Open-source models
- Custom fine-tuned models
- Real-time streaming via unified router
- OpenAI-compatible endpoint
Model-Dependent Support:
Supported Models:
- Llama 3.1 series (8B, 70B, 405B Instruct)
- Llama 3.1 Nemotron Ultra
- Hermes 3 Llama 3.2
- CodeLlama 34B Instruct
- Mistral 7B Instruct v0.3
Unsupported Models:
- DialoGPT variants (treats tools as conversation)
- GPT-2, BERT, RoBERTa variants
- Most pre-2024 models
- Not supported via unified router
- Individual model APIs may support
- Not supported
- Not supported
- Not supported via provider
# Required
HUGGINGFACE_API_KEY=hf_...
# Optional
HUGGINGFACE_MODEL=meta-llama/Llama-3.1-8B-Instruct- Tool calling only on specific models
- No vision/multimodal support
- No PDF processing
- Model quality varies significantly
- Some models require approval/licensing
File: src/lib/providers/litellm.ts
Provider Name: litellm
Default Model: openai/gpt-4o-mini
- Access to 100+ models via proxy
- Unified interface for all providers
- Cost tracking and analytics
- Real-time streaming
- Proxies to underlying provider streams
- Full tool support
- Depends on backend model capabilities
- Depends on backend model
- If proxying to GPT-4o: Vision supported
- If proxying to Gemini: Vision supported
- Varies by configured model
- Not supported via LiteLLM proxy
- Not supported
- JSON schema support
- Type-safe responses
# Required
LITELLM_BASE_URL=http://localhost:4000
LITELLM_API_KEY=sk-anything
# Optional
LITELLM_MODEL=openai/gpt-4o-mini- Requires LiteLLM proxy server running
- Capabilities depend on backend provider
- Model format:
provider/model - Configuration complexity for enterprise setups
File: src/lib/providers/ollama.ts
Provider Name: ollama
Default Model: llama3.1:8b
- Local model execution
- Privacy-first (no data sent to cloud)
- Custom model support
- Real-time streaming
- Dual API mode:
- Native Ollama API (
/api/generate) - OpenAI-compatible API (
/v1/chat/completions)
- Native Ollama API (
- Supported on compatible models
- Llama 3.1+ models
- Gemma 3 models with tool training
Model-Dependent:
- LLaVA models - Vision support
- Gemini models - Vision support
- Llama 3.2 Vision - Vision support
Image Support:
- Up to 10 images (conservative limit)
- Depends on model capabilities
- Not supported
- Not supported
- Limited structured output support
# Optional
OLLAMA_BASE_URL=http://localhost:11434
OLLAMA_MODEL=llama3.1:8b
OLLAMA_TIMEOUT=240000
OLLAMA_OPENAI_COMPATIBLE=false- Local compute requirements
- Model quality varies
- No PDF support
- Vision only on specific models
- Slower inference than cloud providers
File: src/lib/providers/openaiCompatible.ts
Provider Name: openai-compatible
Default Model: Auto-discovered or gpt-3.5-turbo
- Any OpenAI-compatible endpoint
- vLLM, FastChat, LocalAI, etc.
- Custom deployment support
- Real-time streaming
- OpenAI-compatible SSE
- Full tool support
- Depends on backend compatibility
- Depends on backend endpoint
- Auto-discovery not available for capabilities
- Not supported
- Not supported
- JSON schema support
- Type-safe responses
# Required
OPENAI_COMPATIBLE_BASE_URL=https://api.custom.com/v1
OPENAI_COMPATIBLE_API_KEY=...
# Optional
OPENAI_COMPATIBLE_MODEL=model-name # Auto-discovers if not set- Capabilities depend entirely on backend
- No standardized capability detection
- Authentication varies by provider
- Model discovery may fail
File: src/lib/providers/openRouter.ts
Provider Name: openrouter
Default Model: anthropic/claude-3-5-sonnet
- Access to 300+ models from 60+ providers
- Unified API for all models
- Automatic failover
- Cost tracking
- Real-time streaming
- Proxies to underlying provider
Model-Dependent Support:
Supported Models:
- Anthropic Claude models
- OpenAI GPT-4 models
- Google Gemini models
- Mistral Large/Small models
- Meta Llama 3.3, 3.2
Unsupported Models:
- Many older/smaller models
- Check model page for tool support
- Depends on selected model
- GPT-4o, Claude, Gemini support vision
- Check model-specific capabilities
- Not supported via OpenRouter
- Not supported
- JSON schema support
- Type-safe responses
# Required
OPENROUTER_API_KEY=sk-or-...
# Optional
OPENROUTER_MODEL=anthropic/claude-3-5-sonnet
OPENROUTER_REFERER=https://your-app.com
OPENROUTER_APP_NAME=YourApp- Tool support varies by model
- Vision support varies by model
- Credit-based pricing system
- Model availability can change
- No PDF support
- OpenAI - Most reliable, best quality
- Anthropic - Long context, advanced reasoning
- Google Vertex - Enterprise-grade, multi-model
- Anthropic - Best vision + PDF support
- OpenAI - Strong vision, no PDF
- Google AI Studio - Good vision + PDF, free tier
- Anthropic - Most advanced agentic workflows
- OpenAI - Reliable function calling
- Google Vertex - Dual provider (Gemini + Claude)
- Ollama - Fully local, no cloud
- N/A - Only Ollama provides local execution
- Google AI Studio - Free tier available
- OpenRouter - Access to free models
- LiteLLM - Cost tracking, routing
- Anthropic - Native extended thinking
- Google AI Studio - Gemini 2.5+, 3.0 thinking
- Google Vertex - Same as AI Studio
| Provider | Auth Type | Env Vars | Complexity |
|---|---|---|---|
| OpenAI | API Key | OPENAI_API_KEY |
Low |
| Anthropic | API Key | ANTHROPIC_API_KEY |
Low |
| Google AI Studio | API Key | GOOGLE_AI_API_KEY |
Low |
| Google Vertex | Service Account | GOOGLE_APPLICATION_CREDENTIALS |
High |
| Amazon Bedrock | AWS Credentials | AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY |
Medium |
| Amazon SageMaker | AWS Credentials | AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY |
High |
| Azure OpenAI | API Key + Endpoint | AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT |
Medium |
| Mistral | API Key | MISTRAL_API_KEY |
Low |
| HuggingFace | API Key | HUGGINGFACE_API_KEY |
Low |
| LiteLLM | Custom | LITELLM_BASE_URL, LITELLM_API_KEY |
Medium |
| Ollama | None | Optional OLLAMA_BASE_URL |
Low |
| OpenAI Compatible | Custom | OPENAI_COMPATIBLE_BASE_URL, OPENAI_COMPATIBLE_API_KEY |
Medium |
| OpenRouter | API Key | OPENROUTER_API_KEY |
Low |
All providers extend BaseProvider class which provides:
- Unified interface for text generation and streaming
- Tool registration and execution
- Middleware support
- Analytics and telemetry
- Error handling
- Message building for multimodal content
Providers are registered via dynamic imports in ProviderRegistry:
- Avoids circular dependencies
- Lazy loading for better performance
- Clean provider isolation
- Tools registered with
MCPToolRegistry - Provider calls
getAllTools()to get available tools - AI model receives tool definitions
- Model calls tools during generation
- Tool results sent back to model
- Process repeats until completion
- v9.62.0 (May 2026) - Multi-provider voice (TTS/STT/realtime); 21+ providers
- v9.60.0 (April 2026) - Added DeepSeek, NVIDIA NIM, LM Studio, llama.cpp providers
- v9.59.0 - Typed
ModelAccessDeniedError+sdk.checkCredentials() - v9.58.0 -
providerFallbackcallback +modelChainconfig - v9.53.0 - AutoResearch autonomous experiment engine
- v9.52.0 - Per-request and per-instance credentials for all providers
- v8.26.1 (January 2026) - 13 providers (historical)
- v8.26.0 - Added video output types
- v8.25.0 - Gemini 3 support improvements
- v8.24.0 - Enhanced provider capabilities
Next Steps:
- See Provider Comparison Guide for feature matrix
- See Provider Selection Wizard for recommendations
- See API Reference for usage examples