Litellm ishaan march23 - MCP Toolsets + GCP Caching fix

[ishaan-berri]

Litellm ishaan march23 - MCP Toolsets + GCP Caching fix

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

• 50-55 passing tests: main is stable with minor issues.
• 45-49 passing tests: acceptable but needs attention
• <= 40 passing tests: unstable; be careful with your merges and assess the risk.

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

Type

🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test

Changes

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Apr 4, 2026 9:34pm

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 3 committers have signed the CLA.

✅ ishaan-jaff
❌ github-actions[bot]
❌ ishaan-berri
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[gitguardian]

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
29203053	Triggered	Generic Password	59ea60f	.circleci/config.yml	View secret
29375658	Triggered	JSON Web Token	59ea60f	tests/test_litellm/proxy/auth/test_handle_jwt.py	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secrets safely. Learn here the best practices.
3. Revoke and rotate these secrets.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks

---

_{Comparing litellm_ishaan_march23 (9b116e2) with main (08df864)}

[greptile-apps]

Greptile Summary

This PR introduces MCP Toolsets — a named collection of {server_id, tool_name} pairs that can be granted to API keys/teams — alongside a GCP IAM Redis caching fix for async cluster clients.

MCP Toolsets:

• New LiteLLM_MCPToolsetTable Prisma model and migration
• mcp_toolsets: String[] column added to LiteLLM_ObjectPermissionTable for per-key/team grant lists
• New CRUD management endpoints: POST/GET/PUT/DELETE /v1/mcp/toolset
• Toolsets accessible at /toolset/{name}/mcp (explicit prefix) and as a fallback in /{name}/mcp when no server name matches
• A shared ContextVar (_mcp_active_toolset_id) in mcp_context.py carries the active toolset ID server-side; any client-supplied x-mcp-toolset-id header is stripped from the ASGI scope before processing
• resolve_toolset_tool_permissions and get_toolset_by_name_cached use user_api_key_cache (Redis-backed DualCache) to avoid per-request DB hits

GCP IAM fix:

• Moves _generate_gcp_iam_access_token and the new GCPIAMCredentialProvider to litellm/_redis_credential_provider.py
• Replaces the one-time token-at-startup approach with a CredentialProvider that regenerates the IAM token on every new connection, fixing the 1-hour token expiry regression

New findings:

• _ensure_eof done-callback can silently drop the EOF sentinel when asyncio.Queue(maxsize=1024) is full, leaving body_iter() blocked
• dynamic_mcp_route incurs a DB round-trip for every unrecognised server name (first-call cache miss path)
• Inline global_mcp_server_manager imports repeated inside three management handlers instead of using the module-level import

Confidence Score: 4/5

PR is safe to merge with a couple of P2 issues to address first.

The MCP Toolsets feature is well-architected with solid security controls (ContextVar injection prevention, header stripping, explicit grant checks), the GCP caching fix is correct, and test coverage for the new access-control paths is good. Two new P2 findings remain: (1) _ensure_eof can silently drop the EOF sentinel when the bounded queue is full; (2) dynamic_mcp_route falls back to a DB lookup for every unrecognised server name. Prior review threads flagged json.dumps serialization into the JSONB column and inline imports which are also still present.

litellm/proxy/proxy_server.py (_ensure_eof callback and dynamic_mcp_route toolset fallback), litellm/proxy/_experimental/mcp_server/toolset_db.py (json.dumps into JSONB — from prior review)

Important Files Changed

Filename	Overview
litellm/proxy/_experimental/mcp_server/toolset_db.py	New DB helper layer for toolset CRUD. Manually calls json.dumps() on the tools Json field before passing to Prisma (and json.loads() on read), storing a JSON string literal in the JSONB column instead of a native JSON array — already flagged in prior review.
litellm/proxy/_experimental/mcp_server/mcp_context.py	New module introducing _mcp_active_toolset_id ContextVar to avoid circular imports between server.py and mcp_server_manager.py; clean and minimal.
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py	Adds resolve_toolset_tool_permissions, invalidate_toolset_cache, and get_toolset_by_name_cached with Redis-backed caching; direct Prisma query in reload_servers_from_database bypasses get_all_mcp_servers helper (previously flagged).
litellm/proxy/_experimental/mcp_server/server.py	Adds _apply_toolset_scope (access-control + permission override) and _merge_toolset_permissions (union toolset grants into key permissions); strips client-supplied x-mcp-toolset-id header to prevent forgery.
litellm/proxy/proxy_server.py	Adds _stream_mcp_asgi_response helper with bounded queue (maxsize=1024) and toolset_mcp_route; _ensure_eof done-callback can silently fail if queue is full when handler task dies.
litellm/proxy/management_endpoints/mcp_management_endpoints.py	Adds CRUD endpoints for toolsets with correct admin-only write guards and toolset-filtered read for non-admins; cache invalidation on all mutating paths.
litellm/_redis_credential_provider.py	New GCPIAMCredentialProvider correctly refreshes IAM token on every new Redis connection, fixing the 1-hour expiry bug in async cluster clients.
litellm/_redis.py	Replaces one-time IAM token generation with GCPIAMCredentialProvider for async cluster; removes dead code and debug log statements.
litellm/proxy/management_helpers/object_permission_utils.py	Adds _extract_requested_mcp_toolsets and extends validate_key_mcp_servers_against_team to enforce that a key's requested toolsets are a subset of its team's allowed toolsets.
tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_toolset_scope.py	New test file with good unit coverage for _apply_toolset_scope, fetch_mcp_toolsets, and ContextVar injection prevention; all tests use mocks (no real network calls).
litellm-proxy-extras/litellm_proxy_extras/migrations/20260321000000_add_mcp_toolsets/migration.sql	Idempotent CREATE TABLE IF NOT EXISTS and ALTER TABLE ADD COLUMN IF NOT EXISTS; unique index on toolset_name; uses JSONB default consistent with schema.

Sequence Diagram

sequenceDiagram
    participant Client
    participant ProxyServer as proxy_server.py
    participant ContextVar as _mcp_active_toolset_id
    participant Server as server.py
    participant Manager as MCPServerManager
    participant Cache as user_api_key_cache
    participant DB as Prisma DB

    Client->>ProxyServer: GET /toolset/{name}/mcp
    ProxyServer->>Manager: get_toolset_by_name_cached(name)
    Manager->>Cache: async_get_cache(toolset_name:{name})
    alt Cache hit
        Cache-->>Manager: MCPToolset
    else Cache miss
        Manager->>DB: find_first(toolset_name=name)
        DB-->>Manager: row
        Manager->>Cache: async_set_cache(...)
    end
    Manager-->>ProxyServer: toolset (id, name, tools)
    ProxyServer->>ContextVar: set(toolset.toolset_id)
    ProxyServer->>Server: handle_streamable_http_mcp(scope, receive, bridging_send)
    Server->>Server: strip x-mcp-toolset-id header from scope
    Server->>Server: _apply_toolset_scope(auth, toolset_id)
    Note over Server: Non-admin: check mcp_toolsets grant list
    Server->>Manager: resolve_toolset_tool_permissions([toolset_id])
    Manager->>Cache: async_get_cache(toolset_perms:{id})
    alt Cache hit
        Cache-->>Manager: server_id to tools map
    else Cache miss
        Manager->>DB: list_mcp_toolsets(toolset_ids=[id])
        DB-->>Manager: toolsets
        Manager->>Cache: async_set_cache(...)
    end
    Manager-->>Server: tool_permissions dict
    Server->>Server: Update object_permission (mcp_servers, mcp_tool_permissions)
    Server-->>Client: filtered tool list / MCP session
    ProxyServer->>ContextVar: reset(token)

Loading

_{Reviews (3): Last reviewed commit: "fix(tests): update reload_servers_from_d..." | Re-trigger Greptile}

[greptile-apps]

Non-idiomatic JSON serialisation for Prisma Json field

tools is defined as Json in the Prisma schema (tools Json @default("[]")). Prisma's Python client expects a Python dict/list for Json fields and handles serialisation itself. By calling json.dumps(...) first, you are passing a Python string to Prisma, which stores a JSON string literal in the JSONB column (e.g., "[{\"server_id\":\"x\"}]") instead of a JSON array ([{"server_id":"x"}]).

The read path in _toolset_from_row compensates for this with an isinstance(tools, str) branch, so round-trips work. However, the JSONB column stores string values instead of arrays, which means:

1. Direct SQL queries (WHERE tools @> '[{"server_id":"x"}]'::jsonb) will silently return no results.
2. Future Prisma middleware or Prisma validators that type-check Json fields may reject the string.

Pass the Python list directly and let Prisma serialise it:

Suggested change

	async def get_mcp_toolset(
	prisma_client: PrismaClient,
	data_dict["tools"] = data_dict.get("tools", [])

And similarly for the update path in update_mcp_toolset (line 83):

# Before
data_dict["tools"] = json.dumps(data_dict["tools"])
# After
data_dict["tools"] = data_dict["tools"]  # Prisma serialises Json fields

[greptile-apps]

Direct Prisma call bypasses get_all_mcp_servers helper

The previous code used get_all_mcp_servers(prisma_client, approval_status="active"). The new code performs an equivalent query directly via prisma_client.db.litellm_mcpservertable.find_many(), bypassing any logic the helper encapsulates.

Per the CLAUDE.md guideline and rule d7156c05, DB access for proxy operations should go through the established helper functions to avoid pattern drift and make future schema changes easier to track. If get_all_mcp_servers doesn't support the approval_status=None case needed here, consider extending it with an include_no_approval parameter rather than duplicating the query inline.

Rule Used: What: In critical path of request, there should be... (source)

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

[greptile-apps]

Unbounded asyncio.Queue can grow without limit for large streaming responses

body_queue: asyncio.Queue = asyncio.Queue() has no capacity bound. If the ASGI handler produces body chunks faster than the HTTP client consumes them (e.g., slow client, large tool-list response), body_queue will accumulate all chunks in memory before they are yielded.

For an MCP server that streams a large JSON payload, this could exhaust worker memory. Consider using a bounded queue to apply backpressure:

body_queue: asyncio.Queue = asyncio.Queue(maxsize=64)

And in bridging_send, replace await body_queue.put(chunk) so it respects the bound (which put already does — it will await until there is space). The _ensure_eof callback uses put_nowait, which should stay as-is to avoid deadlock in the error path.

[greptile-apps]

Inline imports inside methods (CLAUDE.md / style guide rule)

resolve_toolset_tool_permissions, invalidate_toolset_cache, and get_toolset_by_name_cached all contain inline from … import … statements. Per CLAUDE.md: "Avoid imports within methods — place all imports at the top of the file (module-level). The only exception is avoiding circular imports where absolutely necessary."

from litellm.constants import DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL and from litellm.types.mcp_server.mcp_toolset import MCPToolset are not involved in any circular import and can be safely moved to the module-level imports at the top of mcp_server_manager.py. The from litellm.proxy.proxy_server import prisma_client, user_api_key_cache imports are a known circular-import pattern used elsewhere in this file and are acceptable as inline — but the non-circular ones should be hoisted.

The same pattern also appears in _apply_toolset_scope in server.py (lines 2411–2412): from litellm.proxy._types import LiteLLM_ObjectPermissionTable can be moved to the top of the file.

Context Used: CLAUDE.md (source)

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

To fix unused-import issues, remove the specific unused bindings from the import statement while keeping the ones that are actually used. This preserves existing functionality and simply cleans up dead code.

In this file, the antd import currently brings in Modal, Form, Input, message, Spin, Card, Typography, Space. Based on the CodeQL report and the visible usage (Typography aliasing to AntdText), Card and Space are not used. The best fix is to edit ui/litellm-dashboard/src/components/mcp_tools/MCPToolsetsTab.tsx so that the destructuring import from "antd" no longer includes Card and Space, leaving the rest unchanged. No additional methods, imports, or definitions are needed.

To fix the problem, remove the unused variable so that the code no longer declares AntdText if it is not used anywhere. This removes dead code and satisfies the static analysis rule.

The best minimal fix without changing functionality is to delete the line const { Text: AntdText } = Typography; in ui/litellm-dashboard/src/components/mcp_tools/MCPToolsetsTab.tsx. Keeping the Typography import itself is appropriate, because we cannot be sure from the snippet whether other members of Typography are used further down; however, destructuring Text into AntdText clearly serves no purpose if it’s unused.

No new methods, imports, or definitions are required to implement this change; it is purely a removal of an unused variable declaration.

In general, to fix an unused variable warning, either remove the variable (and any associated dead computation) if it truly isn’t needed, or start using it where it was intended to be used. Here, we see that rawSelected already holds the same value as the “resolved” server ID: const mcpServerId = rawSelected.startsWith("toolset:") ? rawSelected : rawSelected; simplifies to just rawSelected. Since no later code uses mcpServerId, the best non‑intrusive fix is to delete this declaration entirely.

Concretely, in ui/litellm-dashboard/src/components/playground/chat_ui/ChatUI.tsx, within the MCP direct mode block starting around line 592, remove the line that declares mcpServerId. Do not add new behavior or change existing variables; just delete the unused variable and keep the rest of the logic intact.