Add support of custom endpoints or local LLM endpoints

[jagad89]

No description provided.

[basicthinker]

@jagad89 we don't support this out of the box.

The plan is to enable local endpoints if their APIs are compatible with OpenAI's. Then you could create a provider with the base URL set to your endpoint, in a similar way like below, and change the providers of your models in the Settings as well.

Is this meeting your requirement? Could you kindly tell more about what local LLMs you are using and what scenarios you plan to use DevChat for?

In any event, we will let you know when the above feature is ready.

TODOs:

• @yangbobo2021 make sure the Settings are sync'ed with ~/.chat/config.yml and new providers in the config file is shown in Settings.
• @basicthinker write docs for this feature.

[jagad89]

I will try the settings with few custom and local endpoints. I will keep posted in same thread.

Basically we will able to use local LLM over intranet without internet connection.

[runjinz]

The configuration modification entry has been changed. Additionally, if it is a local model, the model configuration file needs to be manually modified. Interface modification is not supported at the moment, but it will be planned for modification along with some customer demands in the next steps.

[trevorstr]

I would like to use this extension with a Local LLM, so I don't have to sign up for a paid service. Ollama is a great way to set up a local endpoint on a server or developer workstation.

https://ollama.com/