Add Ollama

[jtpio]

Fixes #482

Ollama seems to be getting popular for running models locally, and looks like it would be good to have in Jupyter AI by default.

• Add OllamaProvider
• Add OllamaEmbeddingsProvider
• Expand the list of available models
• Add documentation
• Mark as experimental, like for GPT4All?

[jtpio]

I guess it would also be fine to have it as a custom model provider in a separate package, if we don't want to maintain it here in this repo.

[dlqqq]

Jeremy, thank you very much for working on this PR and for keeping up with the open source LLM ecosystem. I lack the time to engage there as much as I would like.

Let me know when this PR is ready, and I will approve & merge it once I run it locally. 👍

[lalanikarim]

Looking forward to this integration.
Is it feasible to have the list of supported models as configurable given that there are numerous finetunes out there that would be viable candidates with Ollama?

[jtpio]

Thanks @dlqqq and @lalanikarim!

Yeah I'll try to finish the PR soon. It was originally opened early to see if there was interest to have built-in support for Ollama in jupyter-ai.

Is it feasible to have the list of supported models as configurable given that there are numerous finetunes out there that would be viable candidates with Ollama?

Currently jupyter_ai seems to have support for allowing and blocking models:

jupyter-ai/packages/jupyter-ai/jupyter_ai/extension.py

Lines 61 to 75 in 642ac53

	allowed_models = List(
	Unicode(),
	default_value=None,
	help="""
	Language models to allow, as a list of global model IDs in the format
	`<provider>:<local-model-id>`. If `None`, all are allowed. Defaults to
	`None`.
	Note: Currently, if `allowed_providers` is also set, then this field is
	ignored. This is subject to change in a future non-major release. Using
	both traits is considered to be undefined behavior at this time.
	""",
	allow_none=True,
	config=True,
	)

Although not sure if that would be enough to use models that have not been added to the list of models in jupyter_ai.

[startakovsky]

🙇 @jtpio excited for this

[lalanikarim]

Since Ollama sports an OpenAI compatible Rest API, and jupyter-ai supports setting a base url to override the default OpenAI url, I created new local models based on models supported by Ollama and named them to match OpenAI models.

$ cat mistral-gpt-4.modelfile
FROM mistral

$ ollama create gpt-4 -f mistral-gpt-4.modelfile

This hack currently allows me to use Ollama with jupyter-ai.

Looking forward to integration to support local models hosted with Ollama.

[triinity2221]

@lalanikarim - I am also replicating a similar setup. Just curious are you able to use the /learn, /generate commands in the chat

[lalanikarim]

@lalanikarim - I am also replicating a similar setup. Just curious are you able to use the /learn, /generate commands in the chat

I haven't had any luck with /generate. I run into pydantic errors.

pydantic.v1.error_wrappers.ValidationError: 1 validation error for Outline
sections
  field required (type=value_error.missing)

I haven't tried /learn yet.

[lalanikarim]

@jtpio I am wondering if it would make sense to provide model name in a free form text field for Ollama models and to require %%ai magic code to include model name for ollama models and limit models to a predefined list.

%%ai ollama:mistral

Thoughts?

[jtpio]

@jtpio I am wondering if it would make sense to provide model name in a free form text field for Ollama models and to require %%ai magic code to include model name for ollama models and limit models to a predefined list.

Yeah I guess given the number of available models it would be very difficult to pick just a few here in the jupyter-ai package. And that would also require updating Jupyter AI regularly, increasing maintenance load.

So yes having a way to let users configure the list of models sounds like a good idea.

[siliconberry]

Since Ollama sports an OpenAI compatible Rest API, and jupyter-ai supports setting a base url to override the default OpenAI url, I created new local models based on models supported by Ollama and named them to match OpenAI models.

$ cat mistral-gpt-4.modelfile
FROM mistral

$ ollama create gpt-4 -f mistral-gpt-4.modelfile

This hack currently allows me to use Ollama with jupyter-ai.

Looking forward to integration to support local models hosted with Ollama.

Hi, Thanks for the hack. Its working. But curious.. how/what did you set the OPENAI API key ? Its working inside the chat box (Jupyternaut), but not inside the jupyter notebook. and its asking for OPEN_API_KEY. can u pl help ?

[Mrjaggu]

Anything update on this pr for adding ollama to jupyter ai..

[jtpio]

I'll try to finish this PR soon, and provide a way to configure the list of models (+ docs).

[dannongruver]

@jtpio: Thank you for this ollama integration Jeremy! Also look forward to configurable models. Let me know where I can buy you a coffee.

[lalanikarim]

Hi, Thanks for the hack. Its working. But curious.. how/what did you set the OPENAI API key ? Its working inside the chat box (Jupyternaut), but not inside the jupyter notebook. and its asking for OPEN_API_KEY. can u pl help ?

@siliconberry You can put any value for OPANAI_API_KEY. It is needed for this hack to work but will be ignored by ollama.

[dannongruver]

@lalanikarim in answer to @siliconberry question, have you gotten both the chat/jupyternaut as well as notebook cell (ie magic %%ai) working with ollama?

I’m struggling now to get notebook working. When I set the OPENAI_API_KEY environment variable to a dummy key (sk-abcd1234), the notebook gives an error indicating that ChatGPT doesn’t accept the key. Jupyternaut chat works fine. It seems like either jupyter_ai’s notebook is not using the config.json like Jupyternaut or ollama does not support all ChatGPT api…? Idk, maybe I’m missing something

[dannongruver]

Nevermind @lalanikarim @siliconberry. The magic %%ai does work.

Note: The magic doesn’t use the config.json but uses environment vars for key and url.

[cboettig]

Just joining the chorus of folks who would be excited to see this, especially given the rate at which new open-weights models are regularly appearing for ollama (llama3, phi3, wizardlm2). ollama's use of local GPU seems a lot smoother too than gpt4all too.

[orkutmuratyilmaz]

I'm so excited for this! 😍😍😍