[Bug]: Streaming broken for Gemini?

[paul-gauthier]

What happened?

litellm.completion() seems to output only 1 token from Gemini when stream=True is provided. Other models work fine. Non-streaming works fine with Gemini.

Relevant log output

import litellm

model = "gemini/gemini-1.5-pro"
messages=[{"role": "user", "content": "What model are you"}]
comp = litellm.completion(model=model, messages=messages, stream=True)
print(comp)
for chunk in comp:
    print(chunk)

# Outputs only 1 token worth of output:

<litellm.utils.CustomStreamWrapper object at 0x1090b4690>
ModelResponse(id='chatcmpl-6d4c29f3-66c0-4155-a065-1f17c2ad7f34', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(content='I', role='assistant', function_call=None, tool_calls=None), logprobs=None)], created=1719346052, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)
ModelResponse(id='chatcmpl-6d4c29f3-66c0-4155-a065-1f17c2ad7f34', choices=[StreamingChoices(finish_reason='stop', index=0, delta=Delta(content=None, role=None, function_call=None, tool_calls=None), logprobs=None)], created=1719346054, model='gemini-1.5-pro', object='chat.completion.chunk', system_fingerprint=None)

Twitter / LinkedIn details

No response

[paul-gauthier]

This appears fixed in the latest litellm v1.40.26.