[Q&A] Response streaming fail 自部署流式返回失效

[magicnov]

🥰 需求描述 | Feature Description

I would like to have a functionality for streaming the response from ChatGPT, so that I don't have to wait for the complete generation of the answer.

🧐 解决方案 | Proposed Solution

OpenAI API supports response streaming

📝 补充信息 | Additional Information

No response

[lobehubbot]

👀 @magicnov

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

[arvinxx]

Please provide more info with your deployment.

By default LobeChat support streaming

[magicnov]

Right now, it is not working by default for me. Do I need to enable it somewhere in the settings?

[arvinxx]

No. It should work by default. Are you using custom api proxy？

[magicnov]

No, i am using OpenAI API with gpt-3.5-turbo model.
How can I provide you with a video demonstrating the issue?

[magicnov]

Screen.Recording.2023-11-30.at.14.30.43.mov

[arvinxx]

Are you using the chat-preview.lobehub.com ？

[magicnov]

No, i deployed my own instance with docker

[zhuozhiyongde]

相同问题。我注意到如果本机部署（localhost）使用或者不使用透明代理本地直连，流式特性均正常，但是一旦上服务器，无论使用 fastAPI 或者使用透明代理都不正常，我推测可能是由于 Nginx 配置反向代理导致的问题？

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Same problem. I noticed that if the local deployment (localhost) uses or does not use a transparent proxy for local direct connection, the streaming characteristics are normal, but once on the server, it does not work properly regardless of using fastAPI or using a transparent proxy. I speculate that it may be due to the Nginx configuration inversion. Issues caused to the agent?

[arvinxx]

If you are deploying with Nginx, please check your conf.

try with:

proxy_set_header Upgrade "";
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_cache off;

or

proxy_buffering off;
proxy_request_buffering off; 

for example:

location / {
    proxy_pass http://127.0.0.1:3210/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header REMOTE-HOST $remote_addr;
    proxy_http_version 1.1;
    proxy_set_header Upgrade "";
    proxy_set_header Connection "upgrade";
    chunked_transfer_encoding off;
    proxy_cache off;

    add_header X-Cache $upstream_cache_status;
 }

Or maybe you can try to google about gpt api nginx stream to find some solutions?

[zhuozhiyongde]

我尝试了，这并不生效 @arvinxx
（话说大佬这么高强度维护吗，真的是用爱发电啊，repect！

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

I tried it and it doesn't work @arvinxx
(Speaking of which, is the boss such a high-intensity maintenance? He really generates electricity with love, remember!

[arvinxx]

我尝试了，这并不生效

那我建议：

1. 直接IP访问，先确认IP下是否流式正常；
2. 如果 IP 流式正常，配置nginx代理后，访问网站不正常，说明是 nginx 的配置问题。如果我提供的不生效，可能你要去问问GPT或者搜一下，怎么样配置才能支持流式请求；
3. 如果 IP 流式不正常，看下是否有用三方代理。如果用了三方代理的，可能要问问看三方代理供应商，应该怎么配置。如果是 openai 的没出现流式，再提个单独的 issue 出来我看看。

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

I tried it and it didn't work

Then I suggest:

1. For direct IP access, first confirm whether the streaming under the IP is normal;
2. If the IP streaming is normal but not normal after configuring the nginx proxy, it means there is a configuration problem with nginx. If what I provided does not take effect, you may want to ask GPT or search for how to configure it to support streaming requests;
3. If the IP streaming is abnormal, check whether a third-party proxy is used. If you use a third-party agent, you may want to ask how to configure the third-party agent part. If the streaming does not appear on OpenAI, please file a separate issue and let me take a look.

[canisminor1990]

@zhuozhiyongde

试下这个配置

proxy_cache off;  # 关闭缓存
proxy_buffering off;  # 关闭代理缓冲
chunked_transfer_encoding on;  # 开启分块传输编码
tcp_nopush on;  # 开启TCP NOPUSH选项，禁止Nagle算法
tcp_nodelay on;  # 开启TCP NODELAY选项，禁止延迟ACK算法
keepalive_timeout 300;  # 设定keep-alive超时时间为65秒

[zhuozhiyongde]

@zhuozhiyongde

试下这个配置

proxy_cache off;  # 关闭缓存
proxy_buffering off;  # 关闭代理缓冲
chunked_transfer_encoding on;  # 开启分块传输编码
tcp_nopush on;  # 开启TCP NOPUSH选项，禁止Nagle算法
tcp_nodelay on;  # 开启TCP NODELAY选项，禁止延迟ACK算法
keepalive_timeout 300;  # 设定keep-alive超时时间为65秒

！这个对我有效，感谢大佬

完整配置如下：

location ^~ / {
    proxy_pass http://127.0.0.1:3210; 
    proxy_set_header Host $host; 
    proxy_set_header X-Real-IP $remote_addr; 
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 
    proxy_set_header REMOTE-HOST $remote_addr; 
    proxy_set_header Upgrade $http_upgrade; 
    proxy_set_header Connection "upgrade"; 
    proxy_http_version 1.1; 
    add_header Cache-Control no-cache; 
    proxy_cache off;  # 关闭缓存
    proxy_buffering off;  # 关闭代理缓冲
    chunked_transfer_encoding on;  # 开启分块传输编码
    tcp_nopush on;  # 开启TCP NOPUSH选项，禁止Nagle算法
    tcp_nodelay on;  # 开启TCP NODELAY选项，禁止延迟ACK算法
    keepalive_timeout 300;  # 设定keep-alive超时时间为65秒
}

[canisminor1990]

@arvinxx 文档可以更新下了

[BQQQQB]

这个是在哪里设置的

[sctm002]

在bt的docker里的反向代理设置在哪里啊？在Nginx 软件里的配置文件里没有这些设置，与docker里的反向代理设置不一样。

[arvinxx]

宝塔的 docker 不负责代理吧。Nginx 配置文件你自己找找呗

[sctm002]

我是用的宝塔的docker管理器构建的，你这个是新建的站点搭建的吧。我也去试试你这种方案。

[sctm002]

我在docker用配置文件：$ docker run -d -p 3210:3210
-e OPENAI_API_KEY=sk-xxxx
-e OPENAI_PROXY_URL=https://api-proxy.com/v1
-e ACCESS_CODE=lobe66
--name lobe-chat
lobehub/lobe-chat
构建的，用IP访问时，流式模式是正常的，用绑定的域名访问就不行了。

[arvinxx]

我是用的宝塔的docker管理器构建的，你这个是新建的站点搭建的吧。我也去试试你这种方案。

就是你用 docker 部署ip访问的没问题呀。我的意思是你要绑定域名的话肯定要建一个 Nginx 的转发。在这里你用反向代理转到 docker ip上就好了。上面提到的配置就是为了解决 steam 你额外还需要配置的东西

[sctm002]

找到配置文件了，在 /www/server/panel/vhost/nginx/*****.info.conf 这个位置，修改后就可以了。

[nelson-yan]

有用！ 感谢！

[CoREse]

答案亲测有效！
有个疑问，这个是纯前端应用，数据按理说是不会发往后端的，那为什么后端的反向代理设置会影响数据的获取啊？有大佬能解个惑吗？感谢！

[arvinxx]

#728

[CoREse]

原来如此！感谢！

[summerKK]

大佬们,我的用了上面的nginx 配置.也不行.但是我curl的是流式. 用自建的lobe-chat就不行,用 chat-preview.lobehub.com 的却可以😂😂😂

我的nginx配置

location / {

        if ($http_authorization != "Bearer ~~~~") {
            return 403;
        }

        proxy_pass http://172.17.0.1:8899;

        # add_header Content-Type text/event-stream;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header REMOTE-HOST $remote_addr;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_http_version 1.1;
        add_header Cache-Control no-cache;
        proxy_cache off;
        proxy_buffering off;
        chunked_transfer_encoding on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 300;
    }

[summerKK]

搞错了,我把nginx配置配置给了 chatGPT的API,配置给 lob-chat的域名就好了.

nginx配置是有用的