New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Gunicorn + Flask + Tensorflow in Docker container doesnt work #2124
Comments
Is your Docker setup limiting the maximum memory available to the container? |
Experiencing the same. I don't think Gunicorn is to blame though. I get the same error when running |
@tlaanemaa can you confirm what @mackdelany says? |
Hey. Sorry for disappearing like that. My setup is limiting Docker's RAM a little bit but the same thing happened even when I removed the limitation. I will try out running the api file without gunicorn and r eport back. Thanks! |
@tlaanemaa any news about it? |
@benoitc Heya |
So I tried running the app without gunicorn in the container and that worked. Works:
Logs:
Doesnt work:
Logs:
|
Also, I've made the repository open so you can poke around if you want. |
Could the problem be that gunicorn is listening to localhost inside a container so it cannot be reached from the outside? |
I dont think so because the flask app was doing the same and that was working. |
what does it say on debug level? |
@tlaanemaa how is your Docker daemon networking configured? Per comment from @CaselIT it seems likely that your client isn't able to reach the Gunicorn port over the Docker network. Can you try starting Gunicorn with the arg |
I dont think the problem lies in the network because it seems, from the logs at least, that the server is not starting at all since it never hits log lines that come after tensorflow import Nevertheless I tried your suggestion but it gives me an error
Log
If you want to try yourself then the comtainer image is available at registry.gitlab.com/tlaanemaa/image-classifier |
@tlaanemaa can you repost your updated |
@javabrett Sure
Dockerfile at the time of posting:
|
what's the full log of docker, can you paste the command line it is finally using? |
Unless it is doing something that can't be foregone during debug of this issue, can you run it without Portainer for now? This works for me, Docker Desktop for Mac 2.1.0.5:
Accepts Please run and post full output and result. |
I tried it and it works now. Thanks alot! Whats interesting now tho is that when I do POST requests then thpose are fast but GET requests are super slow. After a while of doing GET requests these get fast but then POST gets super slow and the worker times out. Once it responds to that POST, POSTs are fast again and GETs are slow. It seems as if it can do one fast and it takes time for it to switch :D these are the logs when GET is fast and POST is slow because worker times out:
|
Also, in some situations the |
The current server in your Docker will be configured single/sync threaded, which will be trivial to make busy/blocking, so it is likely you are seeing that. Try adding some args like |
Thanks @javabrett |
Had the same issue. As far as I can guess from my own logs, it looks like |
Hi! As both the maintainer of gevent and a contributor to this project I can categorically state that gevent and gunicorn work well together. Various libraries may interfere, but that’s not the fault of either gunicorn or gevent. Please open a new issue if that’s not the case for you. Thanks! |
Hello
I have a TensorFlow 2.0 project that has a tiny Flask API in front of it so I can make requests to the model through HTTP calls with data preprocessing already done in the API. I chose Gunicorn to run my Flask/TensorFlow application in a docker container. Sadly tho, the worker process that Gunicorn creates hangs in the container until its killed by Gunicorn. The server never comes up and I canno tmake requests to it. Moreover, the same Gunicorn setup works flawlessly outside docker, in my host machine.
Docker logs (It just hangs there and prints a timeout error after a long time)
dockerfile:
api.py:
inference.py
I've searched for a solution to this for ages and havent managed to come up with anything, any help would be hugely appreciated.
Thanks!
The text was updated successfully, but these errors were encountered: