[Bug]: Ollama: base_url is ignored in Windows

[RobbyCBennett]

Bug Description

The base_url parameter of the Ollama class is ignored on Windows. My script works on Linux but not on Windows.

# On Windows, the host and port of `base_url` are ignored
llama_index.llms.ollama.Ollama(model='llama3', base_url=f'http://localhost:{port}')

As a result of the base_url being ignored, the llama-index library tries to talk to localhost at port 11434. This is why the following exception is thrown.

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=11434)

Version

0.10.36

Steps to Reproduce

1. Use a Windows system
2. Install ollama
3. Use ollama to download the model in my script ollama pull llama3
4. Install the dependencies with pip install -r requirements.txt which are below
5. Run the python script below with python bug.py

requirements.txt

llama-index-core==0.10.36
llama-index-embeddings-ollama==0.1.2
llama-index-llms-ollama==0.1.3

bug.py

#! /usr/bin/env python3


import atexit
import os
import re
import signal
import subprocess
import sys

from llama_index.core import Document, Settings, VectorStoreIndex
from llama_index.embeddings.ollama import OllamaEmbedding
from llama_index.llms.ollama import Ollama


def killSubprocess():
    global _ollama_child
    if '_ollama_child' in globals() and _ollama_child:
        if sys.platform == 'win32':
            _ollama_child.kill()
        else:
            _ollama_child.send_signal(signal.SIGINT)


def createLLM(model: str, temperature: float) -> bool:
    # Set signal handler to kill subprocess
    atexit.register(killSubprocess)

    # Set environment for subprocess
    env = {'OLLAMA_HOST': ':0'}
    if sys.platform == 'win32':
        env['USERPROFILE'] = os.environ.get('USERPROFILE')
        env['SYSTEMROOT'] = os.environ.get('SYSTEMROOT')
    else:
        env['HOME'] = os.environ.get('HOME')

    # Spawn ollama subprocess
    global _ollama_child
    executable = 'ollama' if sys.platform == 'win32' else '/usr/local/bin/ollama'
    _ollama_child = subprocess.Popen(
        [executable, 'serve'],
        env=env,
        stdin=subprocess.DEVNULL,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.PIPE,
    )

    # Wait for ollama to print the port to stderr for some reason
    line = 'line'
    port = None
    while line:
        line = _ollama_child.stderr.readline()
        match = re.search(rb'^.*? msg="Listening on \[::\]:(\d+) ', _ollama_child.stderr.readline())
        if match:
            port = int(match[1])
            break

    # Fail if ollama didn't print the prot
    if port == None:
        print(f'Failed to get a port from {executable}', file=sys.stderr)
        return False

    print('-' * 80) # DEBUG
    print(f'Port opened by ollama: {port}') # DEBUG
    print('-' * 80) # DEBUG

    # Talk to ollama subprocess with llama_index
    llm = Ollama(
        base_url=f'http://localhost:{port}',
        model=model,
        temperature=temperature,
        request_timeout=60.0, # seconds
    )
    Settings.llm = llm
    Settings.embed_model = OllamaEmbedding(model_name=model)

    return True


def main():
    createLLM('llama3', 0.2)
    documents = [
        Document(id_='Taco', text='Tortilla and filling')
    ]
    index = VectorStoreIndex.from_documents(documents)


if __name__ == '__main__':
    main()

Relevant Logs/Tracbacks

--------------------------------------------------------------------------------
Port opened by ollama: 64356
--------------------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\util\connection.py", line 85, in create_connection
    raise err
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\util\connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\connectionpool.py", line 496, in _make_request
    conn.request(
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\connection.py", line 400, in request
    self.endheaders()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py", line 1298, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py", line 1058, in _send_output
    self.send(msg)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\http\client.py", line 996, in send
    self.connect()
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\connection.py", line 238, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x00000224E05FA250>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\urllib3\util\retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/embeddings (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000224E05FA250>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\RobbyBennett\llm\bug.py", line 89, in <module>
    main()
  File "C:\Users\RobbyBennett\llm\bug.py", line 85, in main
    index = VectorStoreIndex.from_documents(documents)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\core\indices\base.py", line 145, in from_documents
    return cls(
           ^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\core\indices\vector_store\base.py", line 75, in __init__
    super().__init__(
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\core\indices\base.py", line 94, in __init__
    index_struct = self.build_index_from_nodes(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\core\indices\vector_store\base.py", line 308, in build_index_from_nodes
    return self._build_index_from_nodes(nodes, **insert_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\core\indices\vector_store\base.py", line 280, in _build_index_from_nodes
    self._add_nodes_to_index(
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\core\indices\vector_store\base.py", line 233, in _add_nodes_to_index
    nodes_batch = self._get_node_with_embedding(nodes_batch, show_progress)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\core\indices\vector_store\base.py", line 141, in _get_node_with_embedding
    id_to_embed_map = embed_nodes(
                      ^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\core\indices\utils.py", line 138, in embed_nodes
    new_embeddings = embed_model.get_text_embedding_batch(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\core\instrumentation\dispatcher.py", line 274, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\core\base\embeddings\base.py", line 341, in get_text_embedding_batch
    embeddings = self._get_text_embeddings(cur_batch)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\embeddings\ollama\base.py", line 64, in _get_text_embeddings
    embeddings = self.get_general_text_embedding(text)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\llama_index\embeddings\ollama\base.py", line 89, in get_general_text_embedding
    response = requests.post(
               ^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\RobbyBennett\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\requests\adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/embeddings (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000224E05FA250>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))

[dosubot]

To address the issue where the base_url parameter of the Ollama class is being ignored on Windows, ensure that you are correctly instantiating the Ollama class with the base_url parameter explicitly set. Here's how you can specify the base_url when creating an instance of the Ollama class:

from llama_index.llms.ollama import Ollama

# Replace "http://your_custom_url:port" with your actual URL and port
ollama_instance = Ollama(base_url="http://your_custom_url:port", model="your_model_name", request_timeout=60.0)

Make sure to replace "http://your_custom_url:port" and "your_model_name" with your actual URL, port, and model name. This should direct the HTTP requests to the specified base_url.

If the issue persists after this adjustment, consider the following steps:

1. Check for Typos: Verify there are no typos in the base_url.
2. Review Proxy Settings: If using a proxy, it might redirect requests to localhost. Check your system and application proxy settings.
3. Debug HTTP Requests: Use tools like Wireshark or Fiddler to trace the HTTP requests and confirm they are sent to the correct URL.
4. Update the Library: Ensure you're using the latest version of the llama-index-llms-ollama library.
5. Check for Overrides: Ensure the base_url isn't being overridden elsewhere in your code or by any frameworks you're using.

Following these steps should help you resolve the issue with the base_url parameter being ignored on Windows. If the problem continues, providing more details about your environment and the steps leading to this issue could help in further diagnosing the problem.

Details

Sources

• llama-index-legacy/llama_index/legacy/llms/ollama.py
• llama-index-integrations/llms/llama-index-llms-ollama/llama_index/llms/ollama/base.py
• llama-index-integrations/tools/llama-index-tools-requests/llama_index/tools/requests/base.py
• llama-index-integrations/indices/llama-index-indices-managed-llama-cloud/llama_index/indices/managed/llama_cloud/base.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[logan-markewich]

I don't think windows would cause this parameter to be ignored

Rather, this seems like some weird interaction with how ports work on windows.

I'd suggest trying http://127.0.0.1 on windows ?

[RobbyCBennett]

Good idea, but that doesn't work either. Even if my URL has nothing to do with localhost like 'http://badurl.com:80', the host and port are both ignored. I always get the exceptions:

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=11434)

and

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=11434)

[kyouens]

Not to muddy the waters, but I may have some version of this same problem. Application runs fine on my local machine (MacOS), but when deployed in a Linux docker container, no matter how I set base_url for Ollama, I get this error:

HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/embeddings (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4aa542e720>: Failed to establish a new connection: [Errno 111] Connection refused'))

Edit: A bit more troubleshooting. I installed my application on a clean Debian VM and I have the same issue, so not docker related. I confirmed that I can connect using curl from the VM to successfully query the Ollama server on the remote host. Somehow, the base_url isn't "sticking". What is very confusing to me is that it seems to work fine on my development machine.