OpenAI API Wrapper

[Ichigo3766]

Feature request

Hi,

I was wondering if it would be possible to have a openai based api.

Motivation

Many projects have been built around openai api something similar to what vllm has and few others inference servers have. If TGI can have this, we can just swap the base url for any projects such as aider and many more and use them without any hastle of changing the code.

https://github.com/paul-gauthier/aider
https://github.com/AntonOsika/gpt-engineer
https://github.com/Significant-Gravitas/Auto-GPT

And many more.

For reference, vllm has a wrapper and text-generation webui has one too.

Your contribution

discuss.

[philschmid]

Hello @BloodSucker99, I am not sure that's possible on the server side since models have different prompts.
So it might make sense to implement this on a client side, which converts the open AI schema (List of dicts) into a single prompt.

[Ichigo3766]

Yea i kind of suspected that doing it on server side would not be possible :(

Any chance you/anyone would be interested in building like a middle man for this? A python wrapper that just sits in middle would be cool

[philschmid]

I had some time and started working on something. I will share the first version here. I would love to get feedback if you are willing to try out.

[Ichigo3766]

Id love to try it out. Also possible to communicate over discord? Would make it much easier :)

[philschmid]

Okay, I rushed out the first version. It is in a package i started called easyllm.

Github: https://github.com/philschmid/easyllm
Documentation: https://philschmid.github.io/easyllm/

The documentation also includes examples for streaming

Example

Install EasyLLM via pip:

pip install easyllm

Then import and start using the clients:

from easyllm.clients import huggingface
from easyllm.prompt_utils import build_llama2_prompt

# helper to build llama2 prompt
huggingface.prompt_builder = build_llama2_prompt

response = huggingface.ChatCompletion.create(
    model="meta-llama/Llama-2-70b-chat-hf",
    messages=[
        {"role": "system", "content": "\nYou are a helpful assistant speaking like a pirate. argh!"},
        {"role": "user", "content": "What is the sun?"},
    ],
      temperature=0.9,
      top_p=0.6,
      max_tokens=256,
)

print(response)

[Ichigo3766]

This is interesting. Could you give me an example of connecting this to the TGI api? There is a model space but that would be loading the model again right? So instead if I am usin TGI which has the model loaded, how would i use its api in here and get a openai api out?

[philschmid]

No, its a client. How would add a Wrapper when you don't know the prompt format on the server side?
It might be possible the write a different server.rs which implements common templating and you could define what you want when starting it, but that's a lot of work.

[Ichigo3766]

Hi! I am a bit confused of what you mean by "you don't know the prompt format on the server side". So there is a wrapper made by langchain:
https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/llms/huggingface_text_gen_inference.py

I was kind of thinking this way but for openai if that makes sense.

[Narsil]

you don't know the prompt format on the server side"

I think what @philschmid meant, is how are you supposed to send a final fully formed id sequence.
TGI doesn't know how model where trained/fine-tuned, so it doesn't know what system prompt or user_prompt is. It expects a single full string, as the langchain wrapper sends.

So the missing step is going from

 messages=[
        {"role": "system", "content": "\nYou are a helpful assistant speaking like a pirate. argh!"},
        {"role": "user", "content": "What is the sun?"},
    ],

To:

[[SYS]\nYou are a helpful assistant speaking like a pirate. argh[/SYS] What is the sun <s>

Which is needed for good results with https://huggingface.co/meta-llama/Llama-2-7b-chat-hf for instance (don't quote me on the prompt I did it from memory)

[paulcx]

I agree with @Narsil point. Some people or projects don't use OpenAI-style prompts. Eventually, all messages will be merged into a single string for input to LLM, limiting flexibility. One possible solution is to create an API template on the server side, allowing users to define their preferred API. However, implementing this approach might require a substantial amount of work and could potentially introduce bugs.

I have a question: Why is the TGI API slightly different from the TGI client SDK? For instance, the parameter 'detail' is ignored in the TGI client source code. Shouldn't they be exactly the same?

[Narsil]

Why is the TGI API slightly different from the TGI client SDK?

I'm not sure what you are referring to. There could be some slight out sync between the Python client and the server, but that's not intentional.

[Narsil]

One possible solution is to create an API template on the server side

That's definitely an option, which I would like with guidance and token healing if we were to do it, they seem to serve the same purpose: extending the querying API in a user defined way. (Both the server user and the actual querying user)

[viniciusarruda]

I've implemented a small wrapper around the chat completions for llama2.
The easyllm from @philschmid seems good, and I've compared it with my implementation for llama2 and it gives the same result!

[paulcx]

Why is the TGI API slightly different from the TGI client SDK?

I'm not sure what you are referring to. There could be some slight out sync between the Python client and the server, but that's not intentional.

here is what I'm referring to here. The request parameters is slight different from ones in API. It's ok but why the 'detail' is manually set to True here