Ollama llava proxy_server_config.yaml and curl request

[fatihyildizhan]

Hello, I want to achieve to run ollama/llava via litellm. I think I am missing something. Can you show me how should I proceed? Thanks a lot.

This curl request is working for ollama. I've tried this but litellm proxy gave an error for images. Also, tried with image_url, base_url etc.

curl http://localhost:11434/api/generate -d '{
  "model": "llava",
  "prompt":"What is in this picture?",
  "images": ["iVBORw0KGgoAAAANSuQmCC"]
}'

This is the llava model on proxy_server_config.yaml

model_list:
 - model_name: llava
   litellm_params:
     model: ollama/llava:latest
     api_base: http://localhost:11434

Non working curl request:

curl 'http://0.0.0.0:8000/completions' \
  -H 'content-type: application/json' \
  --data-raw '{"model":"llava","prompt":"What is in this picture?","stream": true, "url": "iVBORwW5gU/VEEmJTcL4Pz/f7mgoAbYkAAAAAElFTkSuQmCC"}' \
  --compressed

This is a code example but when I send this to the proxy it gives error.
https://docs.litellm.ai/docs/providers/ollama#ollama-vision-models

import litellm

response = litellm.completion(
  model = "ollama/llava",
  messages=[
      {
          "role": "user",
          "content": [
                          {
                              "type": "text",
                              "text": "Whats in this image?"
                          },
                          {
                              "type": "image_url",
                              "image_url": {
                              "url": "iVBORw0KGgoAAAANSUhEUgAAAGC"
                              }
                          }
                      ]
      }
  ],
)
print(response)

The errors I got in different attempts are as follows:

1:
data: b'{"error":"invalid options: url, images"}'

2:
{"error":{"message":"litellm.main.completion() got multiple values for keyword argument 'messages'","type":null,"param":null,"code":500}}

[krrishdholakia]

what's the error? @fatihyildizhan

[fatihyildizhan]

Sorry, forgot to add them

1:
data: b'{"error":"invalid options: url, images"}'

2:
{"error":{"message":"litellm.main.completion() got multiple values for keyword argument 'messages'","type":null,"param":null,"code":500}}

[fatihyildizhan]

I've couldn't find llava proxy server config. Maybe llava is different than text models.

[fatihyildizhan]

Also this gives this error: {"error":{"message":"litellm.main.completion() got multiple values for keyword argument 'messages'","type":null,"param":null,"code":500}}

curl 'http://0.0.0.0:8000/completions' \
  -H 'content-type: application/json' \
  --data-raw '{
    "model": "ollama/llava",
    "prompt": "What is in this picture?",
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Whats in this image?"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "iVBORCccttghyy..."
                    }
                }
            ]
        }
    ]
}'

[Lunik]

Here you are mixing text completion and chat completion call

[fatihyildizhan]

Hello @krrishdholakia I'm sure that you're so busy. I would appreciate it if you could look into this issue when you have the opportunity. How can I send a curl request to ollama/llava via proxy? Thanks.

[fatihyildizhan]

#2201

[Lunik]

Here is the call I have made to LiteLLM API and is working since #2201

curl "http://127.0.0.1:8000/v1/chat/completions" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ollama/llava",
    "messages": [
      { "role": "user", "content": [
        { "type": "text", "text": "Whats in this image?" },
        { "type": "image_url", "image_url": { "url": "iVBORw...SuQmCC" } }
      ]}
    ]
  }'

Here is the response from LiteLLM

{"id":"chatcmpl-1b971af2-77f5-47a8-b06e-7465ae01251a","choices":[{"finish_reason":"stop","index":0,"message":{"content":" The image you've provided is a playful and cute illustration of an animated character that looks like an anthropomorphic sheep. It appears to be waving goodbye or saying hello, with a happy expression on its face. This type of drawing is often found in cartoons and animations where animals are given human-like characteristics for comedic effect. The artwork is stylized, with exaggerated features that are typical of cartoon representations. ","role":"assistant"}}],"created":1709154971,"model":"ollama/llava","object":"chat.completion","system_fingerprint":null,"usage":{"prompt_tokens":1,"completion_tokens":95,"total_tokens":96}}

Here is the Litellm proxy config used

model_list: 
  - model_name: llava
    litellm_params:
      model: ollama/llava
      api_base: "http://localhost:11434"

Litellm was started localy with

litellm --config /tmp/config.yaml --debug --detailed_debug

[fatihyildizhan]

Thank you very much. Now it is working. @Lunik