Enable Custom Models

[enochlev]

Problem statment: There are alot of powerfull LLMs backend with terrible UI and alot of nice front end LLMw with bad/unscalable backend.

This PR will allow to use custom models from a private server hosting openai compatible API

Requirments:

a backend with openAI compatble API
here are some popular ones
FastChat, FastChat is an open platform for training, serving, and evaluating large language model based chatbots. Includes scalability capabilities
vLLM faster/scalable version of hosting LLMs
LMDeploy Less popular, but more faster/scalable version of vLLM
llama-cpp-python, a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
text-generation-webui, the most popular web UI. Supports NVidia CUDA GPU acceleration.
LM Studio, a fully featured local GUI with GPU acceleration on both Windows (NVidia and AMD), and macOS.
ctransformers, a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.

this will make libraries like [llama-gpt]https://github.com/getumbrel/llama-gpt
obsolete as the focus of this project is nice UI, but has a unscallable backend

Here is how to test it

1. host one of the models above and get a working vLLM backend.
   I recommend vLLM due to its powerfull backend
2. Host the server via ngrok

To skip the 2 steps above test using my api server... Ill host it for a week or untill this PR is closed

• API Endpoint: https://major-collie-officially.ngrok-free.app/v1/chat/completions API key EMPTY

3. Enter in your credentials into the app
   

4. Choose a model supported by your API
   

5. Start chating on a local private server!!!

[mratanusarkar]

I tried bettergpt.chat by setting API as:
API Endpoint: https://major-collie-officially.ngrok-free.app/v1/chat/completions and API key as EMPTY
on checking "Use custom API endpoint"

but on clicking "Model" on top, I don't get vicuna-7b-v1.5 in the dropdown.
@enochlev could you help on what I am doing wrong?

Also, a follow-up question on how to use private Models with this UI, without using "api.openapi.com"?

[enochlev]

I closed down the server as I said I would host it for a week... I reopened it up.

here is the error you should get

And my code should allow you to add your own custom model into the text box.

I guess after thinking about it for a while, a better option is to send a request to the openai endpoint and check for existing models and add them into the dropdown instead of having the user manual edit it.

[7flash]

I closed down the server as I said I would host it for a week... I reopened it up.

here is the error you should get

And my code should allow you to add your own custom model into the text box.

I guess after thinking about it for a while, a better option is to send a request to the openai endpoint and check for existing models and add them into the dropdown instead of having the user manual edit it.

I think we need both, requesting /models endpoint to show default dropdown but still allow user to add custom models in case if /models endpoint did not return all of them.