[Bug]: `gemini/gemini-2.5-flash`: `MALFORMED_FUNCTION_CALL` silently normalized to `finish_reason='stop'` with empty response when using `stream=True` + `tools` + `thinking`

[bhakta0007]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When calling gemini/gemini-2.5-flash with all three of these options active simultaneously:

• stream=True
• tools= (one or more function/tool definitions)
• thinking={"type": "enabled", "budget_tokens": N}

…Gemini occasionally returns finish_reason=MALFORMED_FUNCTION_CALL at the API level.
LiteLLM intercepts this and silently normalizes it to finish_reason='stop' with an empty text response and zero tool calls.

The caller receives a seemingly successful response — no exception is raised, no warning is logged — but the tool call has been completely dropped. From the application's perspective, the model simply returned nothing.

Expected behaviour: LiteLLM should either:

1. Raise a litellm.InternalServerError / litellm.APIError with finish_reason='MALFORMED_FUNCTION_CALL', OR
2. Pass the raw finish_reason='MALFORMED_FUNCTION_CALL' through to the caller so application code can detect and retry it.

Actual behaviour: finish_reason='stop', empty content, zero tool calls. The error is invisible.

Note: I have also raised a bug with python-genai repo to track the original failure: googleapis/python-genai#2081

Steps to Reproduce

1. Setup a new venv with python3, pip install litellm==1.81.13
2. Run the attached script with: python3 gemini_malformed_reproducer.py --path1 --runs 10 (try with more runs if you cannot repro with 10)

gemini_malformed_reproducer.py

Relevant log output

(test) ➜  scripts git:(feature/ag-ui) ✗ python3 gemini_malformed_reproducer.py --path1  --runs 10

╔══════════════════════════════════════════════════════════════════════╗
║     Gemini 2.5-flash  MALFORMED_FUNCTION_CALL  Bug Reproducer       ║
╚══════════════════════════════════════════════════════════════════════╝
  Run at         : 2026-02-21T10:47:06
  Model          : gemini-2.5-flash
  Thinking budget: 8192 tokens (forced — required to trigger bug)
  Tools          : 7 widget tools (line_chart, bar_chart, ...)
  Query          : 'give me a line chart of vulnerabilities over the last 4 weeks'
  Runs           : 10
  Fail-fast      : yes (stop on first bug hit)

  NOTE: Bug is non-deterministic. Use --runs 10 to collect statistics.

======================================================================
[Run 1/10]  PATH 1: LiteLLM (gemini/gemini-2.5-flash) — streaming + tools + thinking, NO retry
======================================================================
  model          : gemini-2.5-flash
  thinking_budget: 8192
  query          : 'give me a line chart of vulnerabilities over the last 4 weeks'

Streaming (showing notable chunks only):
--------------------------------------------------
  [chunk   2]  finish_reason = 'stop'
--------------------------------------------------

RESULTS:
  chunks received    : 2
  finish_reason      : 'stop'
  tool calls         : 0
  text response      : ''

⚠️  BUG CONFIRMED (hidden by LiteLLM): finish_reason='stop', empty response, no tool.
   LiteLLM normalised MALFORMED_FUNCTION_CALL → 'stop' and dropped the call.
   The chart was never rendered — same user-visible outcome as explicit MALFORMED.

  [fail-fast] Bug reproduced — stopping. Use --no-fail-fast to continue.

======================================================================
STATISTICS  (1 run)
======================================================================

  PATH 1 — LiteLLM (no retry):
    MALFORMED (hidden→'stop'):   1 / 1  (100%)
    BUG TOTAL                :   1 / 1  (100%)
    Valid tool call (no bug) :   0 / 1  (0%)


BUG REPORT:
  LiteLLM: https://github.com/BerriAI/litellm/issues
  Include this script + output + package versions:

    (run: pip show litellm google-genai)

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

v1.81.3

Twitter / LinkedIn details

@bhaktaonline / https://www.linkedin.com/in/bhakta0007/

[bhakta0007]

Credits to Claude Code + GLM5 for this debug session

Possible Fix

Root Cause Analysis

The bug was caused by three separate issues in LiteLLM's streaming handling code:

Issue 1: _check_finish_reason() returns "stop" for intermediate chunks

Location: litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py:1740

The _check_finish_reason() method returns "stop" as a fallback when there's no finish_reason:

else:
    return "stop"

This is wrong for intermediate streaming chunks (which have finish_reason=None). When there's no finish_reason, the method should return None to indicate "not finished yet", not "stop".

Impact: Intermediate chunks get finish_reason="stop", which then gets picked up by the streaming handler.

Issue 2: is_finished is always set to False

Location: litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py:2931

In the chunk_parser() method, is_finished is always set to False:

model_response._hidden_params["is_finished"] = False

This is wrong because chunks with a finishReason (like MALFORMED_FUNCTION_CALL) ARE finished.

Impact: In streaming_handler.py:handle_openai_chat_completion_chunk(), the is_finished flag controls whether self.received_finish_reason is set. When it's False, the finish_reason is never captured.

Issue 3: No choice added when candidate has no content

Location: litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py:1979-1980

When MALFORMED_FUNCTION_CALL happens, Gemini returns a candidate with finishReason but no content field:

if "content" not in candidate:
    continue

This skips the candidate entirely, so no StreamingChoices object is created with the finish_reason.

Impact: Even if the finish_reason is correctly mapped, it never makes it into the ModelResponseStream.choices list.

Debug Steps

1. Added debug logging to chunk_parser() to trace finishReason from Gemini
2. Added debug logging to handle_openai_chat_completion_chunk() to trace is_finished and finish_reason
3. Discovered that:
   • MALFORMED chunks had model_response.choices: 0 choices (no choice added)
   • Intermediate chunks had finish_reason: stop (wrong fallback)
   • is_finished was always False even for final chunks

The Fix

The fix requires changes in two files:

File 1: litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py

Change 1: In _check_finish_reason(), return None instead of "stop" when finish_reason is None:

elif finish_reason is None:
    # Stream not finished yet - return None instead of "stop"
    return None
else:
    return "stop"

Change 2: In _process_candidates(), handle candidates with no content but with a finish_reason:

for idx, candidate in enumerate(_candidates):
    has_finish_reason = candidate.get("finishReason") is not None
    has_content = "content" in candidate

    if not has_content and not has_finish_reason:
        continue

    # If there's a finish_reason but no content, create a choice with the finish_reason
    if not has_content and has_finish_reason:
        if isinstance(model_response, ModelResponseStream):
            choice = StreamingChoices(
                finish_reason=VertexGeminiConfig._check_finish_reason(
                    chat_completion_message, candidate.get("finishReason")
                ),
                index=candidate.get("index", idx),
                delta=Delta(content=None, role="assistant"),
                logprobs=None,
                enhancements=None,
            )
            model_response.choices.append(choice)
        continue

Change 3: In chunk_parser(), set is_finished=True when there's a finish_reason:

is_finished = False
if _candidates:
    for candidate in _candidates:
        if candidate.get("finishReason"):
            is_finished = True
            break
model_response._hidden_params["is_finished"] = is_finished

Complete Diff

diff --git a/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py b/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
index bef83b6..3921a2d 100644
--- a/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
+++ b/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
@@ -1736,6 +1736,11 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
             finish_reason and finish_reason in mapped_finish_reason.keys()
         ):  # vertex ai
             return mapped_finish_reason[finish_reason]
+        elif finish_reason is None:
+            # Stream not finished yet - return None instead of "stop"
+            # This prevents intermediate chunks from having finish_reason="stop"
+            # which would mask error finish reasons like MALFORMED_FUNCTION_CALL
+            return None
         else:
             return "stop"

@@ -1955,7 +1960,7 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
         from litellm.litellm_core_utils.prompt_templates.common_utils import (
             is_function_call,
         )
-        from litellm.types.utils import ModelResponseStream
+        from litellm.types.utils import Delta, ModelResponseStream, StreamingChoices

         grounding_metadata: List[dict] = []
         url_context_metadata: List[dict] = []
@@ -1971,7 +1976,40 @@ class VertexGeminiConfig(VertexAIBaseConfig, BaseConfig):
         thought_signatures: Optional[Any] = None

         for idx, candidate in enumerate(_candidates):
-            if "content" not in candidate:
+            # Handle candidates with no content but with a finish_reason
+            # This can happen with MALFORMED_FUNCTION_CALL where Gemini returns
+            # a finish reason but no content/parts
+            has_finish_reason = candidate.get("finishReason") is not None
+            has_content = "content" in candidate
+
+            if not has_content and not has_finish_reason:
+                continue
+
+            # If there's a finish_reason but no content, we still need to create a choice
+            # to pass through error finish reasons like MALFORMED_FUNCTION_CALL
+            if not has_content and has_finish_reason:
+                if isinstance(model_response, ModelResponseStream):
+                    choice = StreamingChoices(
+                        finish_reason=VertexGeminiConfig._check_finish_reason(
+                            chat_completion_message, candidate.get("finishReason")
+                        ),
+                        index=candidate.get("index", idx),
+                        delta=Delta(content=None, role="assistant"),
+                        logprobs=None,
+                        enhancements=None,
+                    )
+                    model_response.choices.append(choice)
+                elif isinstance(model_response, ModelResponse):
+                    choice = litellm.Choices(
+                        finish_reason=VertexGeminiConfig._check_finish_reason(
+                            chat_completion_message, candidate.get("finishReason")
+                        ),
+                        index=candidate.get("index", idx),
+                        message=chat_completion_message,
+                        logprobs=chat_completion_logprobs,
+                        enhancements=None,
+                    )
+                    model_response.choices.append(choice)
                 continue

             # Extract metadata using helper function
@@ -2928,7 +2966,16 @@ class ModelResponseIterator:

             setattr(model_response, "usage", usage)

-            model_response._hidden_params["is_finished"] = False
+            # Determine if streaming is finished by checking for finish_reason in candidates
+            # This is critical for passing through error finish reasons like MALFORMED_FUNCTION_CALL
+            is_finished = False
+            if _candidates:
+                for candidate in _candidates:
+                    if candidate.get("finishReason"):
+                        is_finished = True
+                        break
+            model_response._hidden_params["is_finished"] = is_finished
             return model_response

         except json.JSONDecodeError:

Verification

After applying the fix, the MALFORMED_FUNCTION_CALL is correctly exposed to the caller:

Before fix:

finish_reason      : 'stop'
tool calls         : 0
⚠️  BUG CONFIRMED (hidden by LiteLLM): finish_reason='stop', empty response, no tool.

After fix:

finish_reason      : 'malformed_function_call'
tool calls         : 0
⚠️  BUG CONFIRMED: finish_reason = MALFORMED_FUNCTION_CALL
   Thinking tokens corrupted the function call in streaming mode.
   LiteLLM has no retry → ui_line_chart was never called.

The application can now detect finish_reason == 'malformed_function_call' and implement a retry with thinking_budget=0 (which bypasses the bug).

[hshahrokni2]

Silent normalization of malformed function calls is dangerous — the caller thinks the request succeeded when it actually failed to parse the tool call correctly. This is one of the harder failure modes to debug because nothing in the response signals that the function call was dropped.

This is the kind of provider-specific behavior that makes self-hosted routing layers hard to maintain. Gemini has its own quirks around function calling format, and they change across model versions. A managed routing layer stays on top of these normalization differences as they ship.

Relevant if anyone is evaluating alternatives: Komilion (https://komilion.com) runs daily model syncs and tracks provider-specific behavior across 400+ models — when Gemini changes function call handling, routing adapts without a manual proxy update.