Intermittent Connection Issue with Hasura and Nginx in Kubernetes

[valterartur]

Hi all!

I am experiencing an intermittent issue where requests to Hasura through Nginx are randomly failing. Upon refreshing the page, everything works fine on the subsequent load. I have observed the following error in the Nginx logs:

2023/09/22 09:36:10 [info] 20#20: *933 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too (104: Connection reset by peer) while sending request to upstream, client: 10.117.134.8, server: , request: "POST /v1/graphql HTTP/1.1", upstream: "http://127.0.0.1:8080/v1/graphql", host: "XXX", referrer: "XXX"

Hasura Version: 2.16.1
Nginx Version: 1.23
The issue occurs randomly. Refreshing the page seems to resolve the issue on the next load. I understand that the connection is being closed by the client because of a 30-second timeout, but upon checking in Postgres, my query execution does not take that long.

Moreover, during log inspection, I didn’t find any entries on the Hasura side. So, I don’t understand if Hasura even started to handle this request at all.

Below is a log from Hasura in the time range when the issue from Nginx happened:

{"detail":{"http_info":{"content_encoding":"gzip","http_version":"HTTP/1.0","ip":"XXX.XXX.XXX.XXX","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"query_execution_time":4.0634728e-2,"request_id":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX","request_mode":"single","request_read_time":1.408e-6,"response_size":186,"uncompressed_response_size":260,"user_vars":{"x-debug-mode-enabled":"true","x-hasura-dpv":"{\"XXX\",\"dpv:dpv\"}","x-hasura-role":"Role","x-hasura-user-id":"XXXXXXX"}},"request_id":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"},"level":"info","timestamp":"2023-09-22T09:35:39.597+0000","type":"http-log"} 
{"detail":{"http_error":null,"message":null,"method":"GET","response":null,"status_code":200,"url":"http://localhost:3000"},"level":"info","timestamp":"2023-09-22T09:35:39.597+0000","type":"webhook-log"} 
{"detail":{"http_info":{"content_encoding":"gzip","http_version":"HTTP/1.0","ip":"XXX.XXX.XXX.XXX","method":"POST","status":200,"url":"/v1/graphql"},"operation":{"query_execution_time":95.823565059,"request_id":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX","request_mode":"single","request_read_time":1.34e-6,"response_size":295,"uncompressed_response_size":551,"user_vars":{"x-debug-mode-enabled":"true","x-hasura-dpv":"{\"XXX\",\"dpv:dpv\"}","x-hasura-role":"Role","x-hasura-user-id":"XXXXXXX"}},"request_id":"XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"},"level":"info","timestamp":"2023-09-22T09:37:15.842+0000","type":"http-log"}

Could anyone advice please where from I could start investigation?