The tengine proxy request will return a data size of 0kb, but the logging status code is 200, and the user client displays the request as failed. #1852
Comments
|
Add some information to address the above issues At present, it has been found that all agents using different versions of tengine have encountered a request with a 0kb and status code of 200. However, the old agents using nginx have not found this situation. It is currently suspected that it is a problem with the tengine processing process. The business mode of all agents using nginx and tengine is the same. |
|
Could you provide us with a minimal reproducible example? From the tcpdump package log, you can reconstruct the request with curl command. Then use that command to request nginx and try to reproduce |
|
Another potentially valuable suggestion could involve referencing the debug log for nginx/tengine. This would significantly enhance our ability to address the issue effectively. This would greatly speed up the problem addressing process if you could furnish us with the debug log pertaining to the unsuccessful request. |
|
Hello, there are records in the error log: client Backend instance ip closed keepalive connection |
|
The return data of the abnormal curl was replicated using curl. Currently, a request with incomplete transmission has been replicated, and the normal return size is around 40164, while this incomplete request is only 28579 root@VTN-ubuntu:~/wangyuhao# curl http://xxx.xxx.54.34:9999/services/userAPI?wsdl wsdl:types |
|
The data returned by curl normally wsdl:types |
|
tengine access_log & dubug error_log 2023/08/22 09:25:33 [notice] 11298#0: 371440 "^/enterprise" does not match "/services/userAPI", client: ..86.115, server: testra-demo.itrus.com.cn, request: "GET /services/userAPI?wsdl HTTP/1.1", host: "..54.34:9999" |
|
The current problem seems to be that Tengine (nginx) is processing the data returned by Upstream, HTTP_ Due to network reasons, the body data was not fully received, but the status record value still returned 200 instead of 5xx. It is not confirmed whether it is related to the soap protocol used by the application, and similar situations have not been encountered in other HTTP requests. |
|
What is the proxy buffer setting for your tengine specifically? |
|
Hello, the parameter settings in the configuration file are shown in the screenshot. Please check if there are any setting parameters you described, where: |
|
Check your 'proxy_buffering' on nginx1 and CA-nginx |
|
Hello, proxy is not set in nginx1_ Buffering parameter settings, CA nginx, please wait here to check the configuration file |
|
Hello, We have checked that there is no definition of proxy in the CA nginx configuration file_ Parameter settings for buffering. |
|
Hello, are there any other directions for investigation or possible solutions to avoid? Looking forward to your reply |
|
I reproduce this scenario with a small configuration: worker_processes 1;
error_log logs/error.log debug;
events {
worker_connections 1024;
}
stream {
server {
listen 2888;
content_by_lua_block {
ngx.say("HTTP/1.1 200 OK\r\nServer: Tengine\r\nContent-Length: 7777\r\nConnection: keep-alive\r\nkeep-alive: 60\r\n\r\n");
}
}
}
http {
include mime.types;
default_type application/octet-stream;
log_format main '$remote_addr|$status|$remote_user|$time_local|$request|$status|$body_bytes_sent|$upstream_addr|$upstream_status|';
access_log logs/access.log main;
sendfile on;
keepalive_timeout 65;
proxy_http_version 1.1;
proxy_intercept_errors on;
proxy_next_upstream error http_500 http_502 http_503 http_504;
server {
listen 8080;
server_name localhost;
proxy_buffering on;
location / {
proxy_pass http://127.0.0.1:2888;
}
}
}
When the The reason is that when the upstream gets the header, it is sent to the client immediately, and the status is set to 200 at the same time. When the reading of the body fails, call if (!u->header_sent
|| rc == NGX_HTTP_REQUEST_TIME_OUT
|| rc == NGX_HTTP_CLIENT_CLOSED_REQUEST)
{
ngx_http_finalize_request(r, rc);
return;
}
flush = 0;
if (rc >= NGX_HTTP_SPECIAL_RESPONSE) {
rc = NGX_ERROR;
flush = 1;
}
...
if (rc == 0) {
if (ngx_http_upstream_process_trailers(r, u) != NGX_OK) {
ngx_http_finalize_request(r, NGX_ERROR);
return;
}
rc = ngx_http_send_special(r, NGX_HTTP_LAST);
} else if (flush) {
r->keepalive = 0;
rc = ngx_http_send_special(r, NGX_HTTP_FLUSH);
}I tested nginx is the same behavior. And I didn't find a way to set the status without modifying the code. |
Please help to check the influencing factors and see if there is a way to avoid recording the 200HTTP status code. If the data transmission is incomplete, can it return an error in the 5XX status code to avoid not perceiving business anomalies。
Reproduction environment:
jmeter------->nginx1---------->CA-nginx----------->RA----------->CA
jmeter：Simulated user
nginx1：Simulate front-end proxy
CA-nginx1：Simulate backend
Site configuration:
server {
listen 9999;
server_name nginx1Accessing Domain Names for Users;
index index.html index.htm index.jsp index.do;
root html;
location / {
proxy_pass http://Backend CA-nginx domain name;
access_log /usr/local/nginx/logs/testra-demo-local.log main;
proxy_connect_timeout 800ms;
proxy_read_timeout 800ms;
proxy_send_timeout 800ms;
proxy_next_upstream_tries 2;
proxy_next_upstream_timeout 3;
proxy_next_upstream http_504 http_502 error timeout invalid_header non_idempotent ;
}
}
Test steps:
Simulate direct requests from users to the CA agent, resulting in 30% packet loss by the CA nginx agent
Execute on CA nginx: tc qdisc add dev eth0 root netem loss 30%
Test completion:tc qdisc del dev eth0 root netem loss 30% Turn off packet loss
Simulate user visa processing in jmeter for 10 concurrent 300 seconds
Note: The reason for the speed limit is only to make it easier to reproduce the problem, and this type of phenomenon often occurs on the business side without the speed limit.
Packet capture information:
Filter condition: tcp.stream eq 276
The request from the client to nginx1 has a status of 200 returned from nginx1. From packet capturing, it can be seen that when the backend ca-nginx has not yet fully returned data, the tengine on nginx returns data to the client, but does not include response data.
The interaction process from nginx1 to backend ca-nginx, from packet capture, shows that the response part returned by the data is included in the packet, but the data packet was actively dropped by nginx1.
Capture packets on the ca nginx end:
Filter condition: tcp.stream eq 613, and the response information of the package is also complete.
Engine version: engine: 3.0.0
Tengine configuration on nginx1:
7.Normal contract information
The text was updated successfully, but these errors were encountered: