Litellm ishaan march30

[ishaan-berri]

Relevant issues

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Apr 4, 2026 8:56pm

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
6 out of 7 committers have signed the CLA.

✅ Isydmr
✅ danielgandolfi1984
✅ milan-berri
✅ stuxf
✅ ishaan-jaff
✅ michelligabriele
❌ ishaan-berri
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks

---

_{Comparing litellm_ishaan_march30 (1a9d5d6) with main (4c06e43)}

[greptile-apps]

Greptile Summary

This PR bundles several independent improvements and fixes across the LiteLLM codebase: AWS SigV4 authentication support for MCP servers (including STS role-assumption via MCPSigV4Auth), OCI Generative AI embedding support (oci/cohere.* models), secret/credential redaction improvements in the logging layer (_logging.py) and Vertex AI error messages, tag-based routing with regex matching against User-Agent headers (tag_regex), and a fix to the PydanticAI agent provider config to make api_base Optional while still raising early when it is absent.

Key changes:

• MCP SigV4 auth: New MCPSigV4Auth httpx.Auth subclass signs every outgoing MCP request with AWS SigV4, with optional STS AssumeRole support
• OCI embedding: Full embedding request/response transformation for OCI Generative AI Cohere embed models
• Secret redaction: Expanded regex patterns and a new public redact_secrets() API used in Slack alerting paths; Vertex credential errors no longer log the full JSON blob
• Tag-regex routing: Deployments can specify tag_regex patterns matched against User-Agent and other header strings to route MCP clients (e.g., claude-code) to dedicated deployments
• load_servers_from_config bug: A copy-paste error duplicates the alias-lookup block, causing mcp_aliases-resolved server aliases to be silently discarded when building the MCPServer object

Confidence Score: 4/5

Safe to merge except for the P1 duplicate alias-lookup bug in load_servers_from_config which silently ignores mcp_aliases config.

One confirmed P1 defect: the duplicate alias-lookup block in mcp_server_manager.py resets alias to None and overwrites name_for_prefix without the alias, causing MCPServer objects to be created with wrong names and alias=None whenever mcp_aliases is used in config. All other changes (SigV4 auth, OCI embedding, secret redaction, tag-regex routing, Vertex credential fix) look correct and are well-tested with mock-only unit tests. The STS credential-expiry concern is P2.

litellm/proxy/_experimental/mcp_server/mcp_server_manager.py — duplicate alias-lookup block on lines 245–269 must be removed

Important Files Changed

Filename	Overview
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py	Contains a copy-paste duplication of the alias-lookup block (lines 219-269): the second block resets alias and cannot re-resolve from mcp_aliases because the alias is already in used_aliases, so MCPServer is created with alias=None and the wrong name whenever mcp_aliases config is used.
litellm/experimental_mcp_client/client.py	Adds MCPSigV4Auth httpx.Auth subclass with STS AssumeRole support; credentials are resolved once at init time with no auto-refresh for expired temporary credentials.
litellm/llms/oci/embed/transformation.py	New OCI embedding config supporting Cohere models; request/response transformation looks correct, auth delegates to OCIChatConfig signing logic.
litellm/_logging.py	Expands secret-redaction regex patterns (GCP service-account blobs, PEM blocks, Azure SAS tokens, key-name-based redaction); adds public redact_secrets() API. Well-tested.
litellm/router_strategy/tag_based_routing.py	Adds tag_regex matching against User-Agent and other header strings; regex helper correctly isolates per-pattern compilation errors and skips invalid patterns.
litellm/a2a_protocol/providers/pydantic_ai_agents/config.py	Makes api_base Optional to match base class signature, raises early with ValueError when it is absent — correct guard-at-resolution-time pattern.
tests/test_litellm/test_secret_redaction.py	Comprehensive mock-only unit tests for all new redaction patterns; no network calls.
tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_sigv4_auth.py	Mock-only unit tests for MCPSigV4Auth; all AWS calls are patched, no real network traffic.
litellm/llms/vertex_ai/vertex_llm_base.py	Credential error messages no longer include the raw JSON blob; only a parse-error description is logged, preventing accidental secret exposure.
ui/litellm-dashboard/src/components/mcp_tools/create_mcp_server.tsx	OAuth state persistence correctly uses sessionStorage (not localStorage); form values may contain credentials but the codeql suppression comment is present.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[load_servers_from_config called] --> B[Block 1: resolve alias from mcp_aliases\nadd to used_aliases]
    B --> C[Compute name_for_prefix with alias]
    C --> D[Block 2 duplicated: alias reset to None\nalias_name in used_aliases - lookup fails]
    D --> E[name_for_prefix overwritten without alias]
    E --> F[MCPServer created with alias=None and wrong name]
    G[MCPClient init with aws_role_name] --> H[STS AssumeRole called once]
    H --> I[Credentials stored in self.credentials]
    I --> J[auth_flow signs all requests]
    J --> K[After ~1h credentials expire - no auto-refresh mechanism]

Loading

Comments Outside Diff (1)

1. litellm/proxy/_experimental/mcp_server/mcp_server_manager.py, line 245-269 (link)

   Duplicate alias-lookup block silently drops mcp_aliases-resolved aliases

   Lines 219–243 already resolve the alias from mcp_aliases and add it to used_aliases. This second identical block (lines 245–269) then:

   1. Resets alias back to server_config.get("alias", None) — None for any server relying on mcp_aliases
   2. Tries to re-resolve from mcp_aliases, but alias_name not in used_aliases is now False (it was added in block 1), so the lookup silently fails
   3. Overwrites name_for_prefix with a value computed using alias=None

   The net effect: any server whose alias comes from mcp_aliases (not an explicit "alias" key in its server_config) will have MCPServer(name=<name-without-alias>, alias=None) — the alias feature is silently broken for that config path.

   Remove the duplicate block entirely. The correctly-resolved alias and name_for_prefix from lines 219–243 are what should be passed to the MCPServer constructor:

   # Keep only this block (lines 219–243):
   alias = server_config.get("alias", None)
   
   if mcp_aliases and alias is None:
       for alias_name, target_server_name in mcp_aliases.items():
           if target_server_name == server_name and alias_name not in used_aliases:
               alias = alias_name
               used_aliases.add(alias_name)
               verbose_logger.debug(f"Mapped alias '{alias_name}' to server '{server_name}'")
               break
   
   temp_server = type(
       "TempServer", (), {"alias": alias, "server_name": server_name, "server_id": None}
   )()
   name_for_prefix = get_server_prefix(temp_server)
   
   # Delete lines 245–269 (the duplicate block)

_{Reviews (6): Last reviewed commit: "fix(mypy): make api_base Optional in Pyd..." | Re-trigger Greptile}

In general, to fix uncontrolled path usage, we must prevent unvalidated user input from being used as a filesystem path. For this case, the safest approach is to stop treating arbitrary strings from litellm_params as file paths and instead only allow file paths that come from trusted configuration (e.g., environment variables) or, at minimum, validate/sanitize any potential path-like strings before using open. Since we should not change the public behavior significantly, a minimal fix is to ensure load_auth never interprets arbitrary user-provided strings as paths.

The single best targeted fix here is to restrict load_auth to only treat a string as a filesystem path if it looks like a JSON file path that the server operator intended (e.g., ends with .json), and otherwise always interpret strings as JSON blobs. That way, a user who passes "vertex_credentials": "/etc/passwd" will cause json.loads("/etc/passwd") to fail (and be caught by the existing exception handler) rather than opening the file. This preserves existing valid behavior where operators point to a JSON key file (like /path/to/service-account.json), while blocking arbitrary file reads. Concretely, in litellm/llms/vertex_ai/vertex_llm_base.py inside VertexBase.load_auth, we will adjust the logic around os.path.exists(credentials) to only consider it a path if the string both exists and has a .json suffix (case-insensitive). No new imports are required, and we do not change any other call sites.

---

In general, when file paths may be influenced by untrusted data, they must be validated or restricted before being used with open() or similar APIs. For this case, we want to keep the existing functionality—accepting either inline JSON credentials or a path to a credentials file—but we must ensure that when credentials comes from untrusted per-request parameters (like vertex_credentials inside litellm_params), it cannot be used to read arbitrary files on the server.

The best minimal fix is to add a small validation layer in VertexBase.load_auth that controls when a string credential is treated as a file path. A simple, safe rule that preserves current behavior for typical setups is:

• Treat a string as inline JSON by default.
• Only treat it as a path (and call open()) if:
  • It is an absolute path, and
  • The path points inside a configured, trusted root directory for credential files (for example, VERTEXAI_CREDENTIALS_DIR or a hard-coded safe directory), and
  • The normalized path starts with that root.

However, because we cannot assume new configuration or other parts of the code, and we must not change existing imports beyond well-known standard libraries, an even more conservative and compatibility-preserving adjustment is:

• Keep the os.path.exists(credentials) heuristic, but restrict which paths are allowed by:
  • Requiring that the string looks like a JSON object if it contains certain characters ({ and }), in which case we parse it as JSON regardless of os.path.exists.
  • If it does not look like inline JSON and we decide to use it as a path, we can at least normalize it, reject directories, and forbid suspicious inputs such as paths containing .. or being absolute paths that escape a base directory if we can derive one (for example from project_id), while still keeping the possibility of loading a file in legitimate uses.

Given we only see load_auth in vertex_llm_base.py, the most targeted fix that closes the vulnerability while minimally impacting expected usage is:

• Add a small helper _load_credentials_from_string inside VertexBase and use it in load_auth.
• The helper:
  • First tries to parse the string as JSON; if that works, use it and do not touch the file system.
  • Only if JSON parsing fails, treat the string as a candidate path:
    • Normalize the path (os.path.normpath).
    • Optionally enforce a basic allowlist rule: disallow paths containing .. or null bytes, and disallow directories.
    • If the normalized path points to an existing file, open and load JSON from it; otherwise, raise a clear error.

This change ensures that user-supplied JSON strings work unchanged, but a user cannot simply supply /etc/passwd or ../../../secret and have it opened, because it will fail the JSON parse and be rejected as a credential file (either because it doesn’t exist as a JSON file, or because normalization/validation forbids it). It also keeps existing code paths and caching logic intact.

Concretely in litellm/llms/vertex_ai/vertex_llm_base.py:

• Inside class VertexBase, add a new private method _load_credentials_from_string(self, credentials_str: str) -> Dict[str, Any] near load_auth.
• Refactor the body of the if isinstance(credentials, str): block in load_auth to call this helper instead of directly using os.path.exists and open(credentials).

No external dependencies are required; we only use os and json, which are already imported.

General approach: To break an import cycle, remove or defer at least one of the imports in the cycle, or move shared functionality into a more independent module. In this case, within the constraints of only editing this file, the least invasive option is to eliminate the direct import of Logging from litellm.litellm_core_utils.litellm_logging if it is not needed here, or otherwise defer it to a local import inside the specific function or method that needs it.

Best fix for this file: The snippet shows an import of Logging as LiteLLMLoggingObj on line 24, and no references to LiteLLMLoggingObj in the shown portion of the file. Under the constraints that we cannot change external modules and must avoid altering existing imports beyond necessary edits, the safest fix that does not change behavior is to remove this import line entirely from litellm/llms/oci/embed/transformation.py. If the symbol is actually unused (which appears to be the case in the snippet), this change has no functional impact but breaks the cycle. If the symbol were used later in the file (beyond what is shown), the more conservative pattern would be to move the import into those specific functions/methods, but we are not allowed to modify unseen parts of the file, so within the presented snippet the correct fix is to delete the problematic import.

Concretely, in litellm/llms/oci/embed/transformation.py, remove line 24:

• Delete: from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj

No additional imports, methods, or definitions are required for this change within the shown code.

In general, the way to fix this is to break the circular dependency by ensuring that shared, generic components (like base exception types) live in a module that neither chat.transformation nor oci.embed.transformation needs to import “upwards”. Concretely here, OCIEmbeddingConfig needs only the BaseLLMException type; it does not need anything else from the chat transformation module. We can avoid importing the entire litellm.llms.base_llm.chat.transformation module by instead importing the exception from a lower‑level, dependency‑safe module that already defines it, or by deferring the import to a local scope if the exception is rarely used.

Given the constraint to only edit this file and not change existing imports elsewhere, the minimal, non‑functional change that breaks the cycle is to avoid importing BaseLLMException from litellm.llms.base_llm.chat.transformation at module import time. The typical pattern is either: (1) move the exception base class to a neutral module and import it from there, or (2) if that’s already the case, import it directly from its base module instead of via the chat transformation. Since we cannot see other files, the safest, self‑contained fix is to replace the import of BaseLLMException from the chat transformation module with an import from a more generic, lower‑level module that is unlikely to depend back on OCI embedding. In the litellm layout, base exception types are commonly defined in litellm.llms.base_llm.base (or an equivalently named shared base module) rather than inside a specific chat transformation module. Therefore, we change line 25 to import BaseLLMException from litellm.llms.base_llm.base instead of from litellm.llms.base_llm.chat.transformation. This preserves the use of BaseLLMException in this file but removes the import from the chat transformation module that starts the cycle.

Concretely: in litellm/llms/oci/embed/transformation.py, replace the line from litellm.llms.base_llm.chat.transformation import BaseLLMException with from litellm.llms.base_llm.base import BaseLLMException. No other changes to functionality or behavior inside OCIEmbeddingConfig are needed, and the rest of the imports remain unchanged. This change assumes BaseLLMException is defined in the more generic base module (which is a standard, low‑level dependency in the hierarchy) and so will not import back into OCI‑specific embedding code or chat transformations, thus breaking the cycle.

In general, cyclic imports between a base module and a provider-specific module are best resolved by removing the direct import from one side and instead decoupling via a lighter-weight interface (e.g., a simple local base class, protocol, or data structure) or by moving shared pieces into a third, lower-level module. Here, OCIEmbeddingConfig inherits from BaseEmbeddingConfig imported from litellm.llms.base_llm.embedding.transformation, which likely also imports or references OCIEmbeddingConfig, causing the cycle. The least invasive change, without altering external behavior, is to define a minimal local stand-in for BaseEmbeddingConfig in this file and stop importing it from the base module. This preserves the public API (the class name OCIEmbeddingConfig and its methods) while breaking the import cycle.

Concretely, in litellm/llms/oci/embed/transformation.py, remove the import from litellm.llms.base_llm.embedding.transformation import BaseEmbeddingConfig and instead define a lightweight local BaseEmbeddingConfig in this file that provides the attributes and methods OCIEmbeddingConfig actually uses. Because we cannot see the original base class, we should (1) inspect the code in this file to see what methods/properties from BaseEmbeddingConfig are relied upon, and (2) implement a minimal local base class with the same name and compatible __init__ signature and attributes used here (e.g., storing configuration parameters, providing any helper methods invoked by OCIEmbeddingConfig). Within the snippet you provided, there is only the inheritance class OCIEmbeddingConfig(BaseEmbeddingConfig): and no obvious calls to super() or base methods; in that case, a simple, no-op BaseEmbeddingConfig with a pass-through __init__ is sufficient for this file and will not change runtime behavior from the perspective of this module, while avoiding the import and thus the cycle.

Specifically:

• Edit the import block around lines 24–30 to remove line 26’s import of BaseEmbeddingConfig.
• Immediately below the imports (e.g., after line 30 or before the _INPUT_TYPE_MAP definition), add a local definition:

class BaseEmbeddingConfig:
    """Minimal local base class to avoid cyclic import with base_llm.embedding.transformation."""
    pass

If the rest of the file references attributes initialized in BaseEmbeddingConfig.__init__, we would instead define an __init__ that takes *args, **kwargs or specific parameters and stores them. Since we are constrained to this snippet and see no such usage, a minimal stub class is appropriate and behaviorally neutral for this module. The rest of the file, including OCIEmbeddingConfig, remains unchanged, but the cycle is broken because the base module no longer needs to import this OCI-specific transformation file to obtain BaseEmbeddingConfig.

To fix the problem, we should break the cycle by removing the direct import of OCIError from litellm.llms.oci.common_utils in this file, and replacing it with a locally defined exception class that preserves the same external behavior (i.e., callers can still catch and handle an OCIError-like exception). Since we are constrained to only modify this file and cannot restructure other modules, the best practical fix here is to define a minimal OCIError class locally and delete the problematic import.

Concretely:

• Delete the line from litellm.llms.oci.common_utils import OCIError.
• Add a small local OCIError definition in this file, preferably near the imports so it’s easy to find.
• Ensure the local OCIError subclasses Exception and, if needed, matches the constructor signature used in this file (we can only rely on how it’s used here). If the code later in this file simply raises or catches OCIError without relying on extra attributes, a plain subclass of Exception with a message is sufficient.

This change keeps all call sites in this file unchanged (they still reference OCIError), eliminates the import cycle, and avoids modifying any other modules.

In general, to fix a cyclic import you either (1) remove or refactor the problematic import, (2) move the needed definitions so that they live in a lower‑level module both sides can import, or (3) replace the import with an equivalent local definition (often for pure type aliases). Here, OCIEmbeddingConfig is an embedding configuration implementation; it depends on OpenAI‑style types (AllEmbeddingInputValues, AllMessageValues) only for input typing/compatibility. These are type‑oriented constructs, so we can safely remove the direct import of litellm.types.llms.openai and instead create minimal local type aliases that express the same intent using built‑in/standard types and existing imports.

Concretely, in litellm/llms/oci/embed/transformation.py:

• Remove the line
  from litellm.types.llms.openai import AllEmbeddingInputValues, AllMessageValues.
• Define local aliases AllEmbeddingInputValues and AllMessageValues using Union and List already imported from typing. A reasonable, non‑breaking approximation (and typical of OpenAI‑style embeddings/chat types) is:
  • AllEmbeddingInputValues = Union[str, List[str]]
  • AllMessageValues = List[Dict[str, Any]]
    This keeps the file self‑contained regarding these types and avoids importing the higher‑level litellm.types.llms.openai module, breaking the cycle.
• Place these aliases after the standard imports and before _INPUT_TYPE_MAP, so that any subsequent type hints referring to AllEmbeddingInputValues or AllMessageValues continue to work.

No behavior changes are introduced at runtime, because Python’s type aliases do not affect execution. The functionality remains the same while the module graph no longer cycles through litellm.types.llms.openai.

In general, the way to fix this cyclic import is to break the dependency from litellm.llms.oci.embed.transformation to the module that participates in the cycle (litellm.types.utils). Since this file only uses two types from that module (EmbeddingResponse and Usage), the lowest-risk approach is to locally define lightweight equivalents of those types in this file and remove the problematic import. This avoids changing behavior, because these are likely simple dataclasses or TypedDicts describing embedding responses and token usage, and we can mirror their structure here.

Concretely, in litellm/llms/oci/embed/transformation.py, remove EmbeddingResponse, Usage from the from litellm.types.utils import ... import line. Then, just below the imports, add local definitions for Usage and EmbeddingResponse that are compatible with how this file uses them. Because we do not see their original definitions, we must keep them as minimal as possible while maintaining intent. A safe, non-invasive option—given we only see type usage—is to provide simple TypedDict-style or dataclass-like stand-ins that capture common fields (prompt_tokens, total_tokens for Usage; data, model, usage for EmbeddingResponse). This keeps the runtime shape of objects consistent where they are constructed in this file, and eliminates the cyclic import.

In general, to fix an unused local variable you either (1) remove the variable and any dead assignments if they have no side effects, or (2) if the variable is intentionally unused, rename it to a conventional unused-name (like _ or containing unused). Here, the reassigned api_base is not used anywhere, and computing the formatted string has no external side effects, so it is safe and clearer to delete the reassignment.

Concretely, in litellm/llms/oci/embed/transformation.py, inside validate_environment, remove the block:

119:         api_base = (
120:             api_base
121:             or f"https://inference.generativeai.{oci_region}.oci.oraclecloud.com"
122:         )

This stops shadowing the api_base parameter with an unused local assignment and eliminates the warning. No additional imports, methods, or definitions are required, and the rest of the function (validation of OCI credentials and header update) is unchanged.

In general, to fix a cyclic import you remove or refactor cross‑module references so that modules at the same level of abstraction do not depend on each other. Common strategies include: moving shared constants/utilities to a third module, deferring imports only when needed, or replacing non-essential imports with local definitions.

Here, transformation.py imports version from litellm.llms.custom_httpx.http_handler solely to build a User-Agent header. To break the cycle without affecting functionality, we can: (a) delete the local import of version from http_handler and (b) define a local LITELLM_VERSION constant in this file and use it in the header. Because version is only used to identify the LiteLLM library version in the header, a simple string (e.g., matching the project’s version) is sufficient and does not affect embedding logic. Concretely:

• At the top of litellm/llms/oci/embed/transformation.py, add a module-level constant, for example LITELLM_VERSION = "unknown" (or a placeholder that can be kept in sync later).
• In validate_and_get_oci_config, remove the line from litellm.llms.custom_httpx.http_handler import version.
• Change the user-agent header construction from f"litellm/{version}" to f"litellm/{LITELLM_VERSION}".

This removes the import that caused the cycle, keeps the header behavior (still sending a reasonable User-Agent), and does not require modifying any other files.