Litellm ishaan april6

[ishaan-berri]

Relevant issues

Changes

• fix: batch-limit stale managed object cleanup to prevent 300K row UPDATE (fix: batch-limit stale managed object cleanup to prevent 300K row UPDATE #25227)
• [bug-fix] return actual status code - /v1/messages/count_tokens endpoint ([bug-fix]return actual status code - /v1/messages/count_tokens endpoint #21352)
• feat(teams): per-member model scope + team default_team_member_models (feat(teams): per-member model scope + team default_team_member_models #24950)
• feat: multiple concurrent budget windows per API key and team (feat: multiple concurrent budget windows per API key and team #24883) (feat: multiple concurrent budget windows per API key and team (#24883) #25109)
• chore: sync schema.prisma copies from root
• fix(types): close unclosed paren on budget_limits field in TeamBase

Pre-Submission checklist

• Added testing in tests/test_litellm/
• PR passes all unit tests on make test-unit

CI (LiteLLM team)

• Branch creation CI run
  Link:

• CI run for the last commit
  Link:

• Merge / cherry-pick CI run
  Links:

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Apr 17, 2026 11:01pm

[codspeed-hq]

Merging this PR will not alter performance

✅ 16 untouched benchmarks

---

_{Comparing litellm_ishaan_april6 (e5824b1) with main (5cb9b08)}

[greptile-apps]

Greptile Summary

This PR bundles four features: multi-window concurrent budget enforcement for keys and teams, per-team-member model scoping with a default_team_member_models team default, a fix to the /v1/messages/count_tokens status-code propagation, and a batch-limit guard on the stale-object cleanup job. The schema migrations are additive and safe, the new auth checks reuse the existing get_team_membership cache correctly, and the _upsert_budget_and_membership refactor to in-place updates is a genuine bug fix.

• P1 – Wrong field names in budget_limits docstring examples: four endpoint docstrings document budget_limit/time_period but the BudgetLimitEntry model requires max_budget/budget_duration; callers following the docs receive a 422 immediately.
• P2 – Window reset_at always re-anchored on update: _set_budget_reset_at and prepare_key_update_data call get_budget_reset_time() for every window on every update, so editing a key or team mid-period silently extends existing windows instead of preserving the original start time.
• P2 – N+1 writes in reset_budget_windows: the keys and teams loops each issue one update DB call per changed record; these could be issued concurrently with asyncio.gather.

Confidence Score: 4/5

Safe to merge after fixing the budget_limits docstring field names — wrong examples cause 422 for all API callers trying the feature.

Two P1 docstring errors (same root cause, different endpoints) mean anyone trying budget_limits via the documented API will get a 422 until they independently discover the correct field names. All other findings are P2 (N+1 background writes, window-timer extension on update). Core auth logic, migrations, and the new Pydantic types look correct.

litellm/proxy/management_endpoints/key_management_endpoints.py (lines 1223, 2157) and litellm/proxy/management_endpoints/team_endpoints.py (line 832) for the docstring fixes; litellm/proxy/common_utils/reset_budget_job.py for the timezone and N+1 write issues.

Important Files Changed

Filename	Overview
litellm/proxy/auth/auth_checks.py	Adds multi-window budget enforcement and per-member model scope checks to the critical auth path; both use the existing get_team_membership cache so no net new DB queries for warm caches.
litellm/proxy/common_utils/reset_budget_job.py	New reset_budget_windows method issues N+1 DB writes for changed keys/teams; timezone-stripping in _reset_expired_window can cause stale windows for non-UTC deployments (flagged in prior review thread).
litellm/proxy/management_endpoints/key_management_endpoints.py	Adds budget_limits to key generation/update; docstring examples use wrong field names (budget_limit/time_period) that will cause 422 errors for API callers; window reset timestamps are not preserved across updates.
litellm/proxy/management_endpoints/team_endpoints.py	Adds budget_limits and default_team_member_models to team create/update; docstring examples have wrong field names; _set_budget_reset_at resets all window timers on every update rather than preserving existing ones.
litellm/proxy/management_endpoints/common_utils.py	Correctly refactors _upsert_budget_and_membership to update existing budgets in-place rather than always creating new records; adds allowed_models field support.
litellm/proxy/proxy_server.py	Adds per-window spend counter increments for multi-budget keys and teams using cache lookups; extracts _try_provider_token_count helper that correctly propagates HTTP status codes from provider token-counting APIs.
litellm/proxy/_types.py	Adds BudgetLimitEntry, budget_limits to key/team types, allowed_models to LiteLLM_BudgetTable, and default_team_member_models to TeamBase; closes unclosed paren in TeamBase.budget_limits.
tests/test_litellm/proxy/auth/test_multi_budget_windows.py	New unit tests for multi-window budget enforcement; all mock-based, no real network calls; covers under-budget, over-first-window, over-second-window, and Pydantic object coercion cases.
tests/test_litellm/proxy/common_utils/test_upsert_budget_membership.py	Tests updated to reflect the new in-place update behavior for existing budget IDs; assertions correctly flip from create to update calls.

Sequence Diagram

sequenceDiagram
    participant Client
    participant common_checks
    participant auth_checks
    participant Cache as UserKeyCache
    participant DB as PrismaDB
    participant SpendCounter

    Client->>common_checks: request(model, team, virtualkey)
    common_checks->>auth_checks: _check_team_member_model_access
    auth_checks->>Cache: get_team_membership(user_id, team_id)
    alt cache miss
        Cache->>DB: find_unique litellm_teammembership
        DB-->>Cache: membership + allowed_models
    end
    Cache-->>auth_checks: membership
    auth_checks-->>common_checks: 401 if model not allowed

    common_checks->>auth_checks: _team_multi_budget_check
    auth_checks->>SpendCounter: get_current_spend(team:window)
    auth_checks-->>common_checks: 429 if window exceeded

    common_checks->>auth_checks: _virtual_key_multi_budget_check
    auth_checks->>SpendCounter: get_current_spend(key:window)
    auth_checks-->>common_checks: 429 if window exceeded

    common_checks-->>Client: allowed

    Note over SpendCounter: Post-response spend tracking
    Client->>SpendCounter: increment_spend_counters(cost)
    SpendCounter->>SpendCounter: incr spend:key (global)
    SpendCounter->>SpendCounter: incr spend:key:window (per BudgetLimitEntry)
    SpendCounter->>SpendCounter: incr spend:team (global)
    SpendCounter->>SpendCounter: incr spend:team:window (per BudgetLimitEntry)

Loading

_{Reviews (5): Last reviewed commit: "style: run black formatter on files from..." | Re-trigger Greptile}

[greptile-apps]

Timezone stripping causes incorrect window reset timing for non-UTC deployments

_reset_expired_window strips timezone info from the stored reset_at string with .replace(tzinfo=None), then compares against datetime.utcnow() (also naive). When litellm_settings.timezone is configured to a non-UTC zone (e.g. America/New_York, UTC−5), get_budget_reset_time returns a TZ-aware datetime whose .isoformat() looks like 2026-04-07T05:00:00-05:00. After stripping the offset, the naive value 2026-04-07T05:00:00 is compared against UTC now (2026-04-07T10:00:00) — making 05:00 > 10:00 → False — so the window is not reset even though it expired 5 hours ago.

A safer approach keeps both sides timezone-aware:

from datetime import timezone

now = datetime.now(timezone.utc)
reset_at_str = window.get("reset_at")
if not reset_at_str:
    return False
reset_at = datetime.fromisoformat(reset_at_str.replace("Z", "+00:00"))
# reset_at is timezone-aware; compare directly without stripping
if reset_at > now:
    return False

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 4 committers have signed the CLA.

✅ otaviofbrito
✅ yuneng-berri
❌ github-actions[bot]
❌ ishaan-berri
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

❌ Patch coverage is 83.90023% with 71 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/evals/main.py	0.00%	11 Missing ⚠️
litellm/llms/custom_httpx/llm_http_handler.py	60.71%	11 Missing ⚠️
...llm/litellm_core_utils/prompt_templates/factory.py	42.85%	4 Missing ⚠️
litellm/llms/bedrock/chat/invoke_handler.py	0.00%	4 Missing ⚠️
...litellm_responses_transformation/transformation.py	33.33%	2 Missing ⚠️
...tellm/integrations/SlackAlerting/slack_alerting.py	33.33%	2 Missing ⚠️
litellm/llms/anthropic/chat/handler.py	71.42%	2 Missing ⚠️
litellm/llms/anthropic/common_utils.py	92.30%	2 Missing ⚠️
litellm/llms/bedrock/realtime/handler.py	33.33%	2 Missing ⚠️
litellm/llms/gemini/realtime/transformation.py	33.33%	2 Missing ⚠️
... and 26 more

📢 Thoughts on this report? Let us know!