-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
配置模型时,可以配置在别的服务器部署的chatglm6b吗 #4012
Comments
Yes, you can configure Langchain Chatchat to use a ChatGLM6B model deployed on a different server. Here's how to do it:
ONLINE_LLM_MODEL = {
"chatglm3-6b-remote": {
"api_base_url": "https://your.remote.server/api",
"api_key": "your_api_key_here",
"provider": "YourProviderWorker",
},
}
import httpx
async def get_ChatOpenAI(model_name: str, temperature: float, max_tokens: Optional[int], callbacks: list):
api_endpoint = "http://localhost:5000/generate"
data = {
"model_name": model_name,
"temperature": temperature,
"max_tokens": max_tokens,
}
async with httpx.AsyncClient() as client:
response = await client.post(api_endpoint, json=data)
generated_text = response.json().get("text", "")
return generated_text
Remember, network latency and bandwidth are crucial factors to consider for maintaining responsive interactions with the remote model. For more detailed examples and discussions on similar setups, you might find these threads helpful:
|
大佬们 请问下
目前看到文档中,可以配置本地大模型和在线大模型api
但是本地大模型似乎要在一台机器上
langchain chatchat 可以配置部署在另外一台机器上的大模型chatglm6b吗
The text was updated successfully, but these errors were encountered: