Rocket Production Freezes. #1302
Comments
|
seeing this same symptom. when i strace the stuck process, i see it's in futex: i consistently see this after running for a few days, the repo is https://github.com/wesleyac/hanabi. |
|
First, try disabling keep-alive (in Rocket.toml or with |
|
Thanks, I change that variable. We will see how it behaves and I will leave feedback here. |
|
this appears to have fixed the issue for me, although i should probably wait another ~week or so to say for sure. |
|
If disabling keep-alive is a fix, there are two main possibilities that come to mind:
|
|
FWIW, my environment is that I have rocket running on a single thread behind nginx as a reverse proxy. If there's a way for me to get more info on what's happening or to test these hypotheses, let me know. the fact that i have a single worker thread could explain why i was seeing this more often than @migeruj |
|
One worker thread can only serve a single request at a time, which could easily explain seeing it more often - especially if a handler is waiting to send/receive data. |
|
In my case I am also doing a reverse proxy with nginx so I am going to check the logs and do load tests. Indeed I have 8 workers so I see this situation weeks apart. |
|
I experience the same issue. Setup:
It ran for about 4 weeks before freezing, restarting the docker container fixed it. |
|
every now and then i get a notification on this issue and hope there's a solution already alas, there still isn't so today i was musing whether we could put rocket behind a WAF like ModSecurity, when i realised: a properly configured proxy server should answer half of these requests with and while that doesn't solve the issue, it does leave rocket with more threads to answer real requests, instead of freezing up on some automated attacks, that aren't even targeting your app! |
|
My guess is that this is simply #1434 with poor keep-alive behavior and a buggy |
Hi guys, I have a development of a web page with Rocket, I use tera templates for views. I have a problem with some robots that visit the page and access Paths. I use docker to orchestrate the container of the app.
The problem is that after so many wrong visits. The page does not respond. The request exits the load balancer correctly. But it is as if Rocket is left in a frozen state. The only way to keep it running is to restart the container.
What can I do to verify this situation?
Bug Reports
The container keep runing but not recieve request.
Bug reports must include:
Rocket = 0.4.4
This happens on Ubuntu 18.04 (Docker)
A brief description of the bug that includes:
* Just Container Freezes and return timeout.
* After 3-4 weeks of working perfectly. It freezes.
* I expect if is there a error, the container restarts with the restart clauses but keeps running
How you uncovered the bug. Short, reproducible tests are especially useful.
I checked that it wasn't a load balancer problem. I have discarded cpu / ram bottlenecks on the instances, there are no memory leaks bothering the container. Rocket just freeze. The request reaches the load balancer but rocket returns nothing.
Ideas, if any, about what Rocket is doing incorrectly.
Well, actually i dont know if is there something we are doing incorrectly or rocket.
Almost all the logs that were received before going into a freeze state are from robots.
Logs Output
The text was updated successfully, but these errors were encountered: