[Feature]: Intel Arc GPU support

[itlackey]

The Feature

It would be great to have installer support for Intel Arc GPUs or instructions on how to install on Linux with Intel Arc.

Motivation, pitch

The Intel Arc 770 model is an inexpensive GPUs with 16GB of VRAM. It is a good entry level option for people getting into ML. It would be good to support this vendor to expand the supported user base of the application.

Twitter / LinkedIn details

@itlackey

[ishaan-jaff]

@itlackey did you try LiteLLM on a Intel Arc GPU? did you face an error ?

[itlackey]

I have not tried LiteLLM directly but have tried via Open Interpreter which appears to use LiteLLM.

It looks like I may just need to recompile llama.cpp with the CLBLAST support to enable the Intel GPU. However I think it may require a code change to push the model to the XPU.
I am up to try it if you can provide any guidance on where to start.

[ishaan-jaff]

OI does not use LiteLLM for calling local models

this is the project you're looking for @itlackey https://github.com/abetlen/llama-cpp-python

[itlackey]

@krrishdholakia I have been able to get llama.cpp working with my Arc and am currently using it with FastChat to host an OpenAI API endpoint. https://github.com/itlackey/ipex-arc-fastchat

However, the code calling the model typically requires some modification to leverage the Intel PyTorch extensions.
https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/examples.html

Is that not the case in this scenario and I simply need to recompile llama.cpp?

@ishaan-jaff Now that I look at this again, you are correct. I had merged two different issues I am having. I also cannot get OI to connect to my FastChat endpoint that is hosting an OpenAI endpoint locally. I believe that is the path that is going through LiteLLM.

https://github.com/KillianLucas/open-interpreter/blob/main/interpreter/llm/setup_text_llm.py

@itlackey can you share the stacktrace you're seeing?

[itlackey]

Here is the tail of it

File "/usr/local/lib/python3.10/dist-packages/litellm/utils.py", line 2684, in exception_type
raise APIError(status_code=500, message=str(original_exception), llm_provider=custom_llm_provider, model=model)
litellm.exceptions.APIError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. E.g. For 'Huggingface' inference endpoints pass in completion(model='huggingface/lmsys/vicuna-7b-v1.3',..) Learn more:...

However after digging a bit more this does seem to be more of an implementation issue with interpreter (or user error) than an issue with LiteLLM. I posted an issue in their repo to see what guidance they could provide. It has some additional info including the full stack trace.

openinterpreter/open-interpreter#552

did you pass in model as `lmsys/vicuna-7b-v1.3` or `huggingface/lmsys/vicuna-7b-v1.3`?

…

On Thu, Sep 28, 2023 at 2:32 PM IT Lackey ***@***.***> wrote: Here is the tail of it File "/usr/local/lib/python3.10/dist-packages/litellm/utils.py", line 2684, in exception_type raise APIError(status_code=500, message=str(original_exception), llm_provider=custom_llm_provider, model=model) litellm.exceptions.APIError: LLM Provider NOT provided. Pass in the LLM provider you are trying to call. E.g. For 'Huggingface' inference endpoints pass in completion(model='huggingface/lmsys/vicuna-7b-v1.3',..) Learn more:... However after digging a bit more this does seem to be more of an implementation issue with interpreter (or user error) than an issue with LiteLLM. I posted an issue in their repo to see what guidance they could provide. It has some additional info including the full stack trace. openinterpreter/open-interpreter#552 <openinterpreter/open-interpreter#552> — Reply to this email directly, view it on GitHub <#481 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEF7LKZIN7Z2DLHN5WWTMLTX4XUFLANCNFSM6AAAAAA5LCTFMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[itlackey]

I have tried both ways. The above is without the huggingface prefix. Below are examples of including the prefix.

Here are some examples:
root@dae1ccdf5359:/apps/open-interpreter# interpreter -y -ab http://172.19.0.2:8000/v1 -m huggingface/lmsys/vicuna-7b-v1.3

test

We were unable to determine the context window of this model. Defaulting to 3000.
If your model can handle more, run interpreter --context_window {token limit} or interpreter.context_window = {token limit}.
Also, please set max_tokens: interpreter --max_tokens {max tokens per response} or interpreter.max_tokens = {max tokens per response}

^Croot@dae1ccdf5359:/apps/open-interpreter# interpreter -y -ab http://172.19.0.2:8000/v1/completion -m huggingface/lmsys/vicuna-7b-v1.3
test

We were unable to determine the context window of this model. Defaulting to 3000.
If your model can handle more, run interpreter --context_window {token limit} or interpreter.context_window = {token limit}.
Also, please set max_tokens: interpreter --max_tokens {max tokens per response} or interpreter.max_tokens = {max tokens per response}

^Croot@dae1ccdf5359:/apps/open-interpreter# interpreter -y -ab http://172.19.0.2:8000 -m huggingface/lmsys/vicuna-7b-v1.3
test

We were unable to determine the context window of this model. Defaulting to 3000.
If your model can handle more, run interpreter --context_window {token limit} or interpreter.context_window = {token limit}.
Also, please set max_tokens: interpreter --max_tokens {max tokens per response} or interpreter.max_tokens = {max tokens per response}

---

This is how I am starting FastChat

python3 -m fastchat.serve.controller --host 0.0.0.0

python3 -m fastchat.serve.model_worker --device xpu --host 0.0.0.0 --max-gpu-memory 14Gib --num-gpus 1 --gpus 0 --model-path lmsys/vicuna-7b-v1.3

python3 -m fastchat.serve.openai_api_server --host 0.0.0.0 --port 8000

---

If I use the custom/ prefix I get a JSON out of range error.
Logs

[itlackey]

I apologize, this is user error when using Open Interpreter with a FastChat API.

The correct command is

interpreter -y -ab http://172.19.0.2:8000/v1 -m openai/vicuna-7b-v1.3

I really appreciate all of the help trying to figure out that issue!

It would still be nice to have native Intel support with LiteLLM and Open CI. I may investigate using LiteLLM directly with my GPU as soon as I get time. With that said, I am a web dev with very little Python experience so far. So that PR might take a minute.

sounds good - closing as this seems resolved then

[itlackey]

Can we leave it open for now? I would like to get Arc support working so FastChat isn't needed as Middleware.

Also OI only works for one message and the crashes with FastChat.

that's fair. what're next steps then? @itlackey

My understanding is that this is a local model issue - is there something specific litellm can help with here? OI doesn't use us for local model calls -- we handle API calls

[itlackey]

To enable Arc I believe I need to install the instwll drivers and toolkit, compile llama.cpp with CLBlast enabled, and possibly make a small change to the codebase to send the model to the Intel GPU.

Unfortunately the Intel extensions for PyTorch require an import and a call to to_xpu to get the model to be processed via the GPU. I have a docker container published that does the installs and llama.cpp compiling. As soon as I get a chance, I will try to get LiteLLM running on that container. If I can make it work, I will put together a PR.
If I can't, I'll report back.

If you're interested here is the container I use to host FastChat with Intel support.
https://github.com/itlackey/ipex-arc-fastchat