Gunicorn eats more memory in django #1250
Comments
|
you maybe need to decrease the workers, try this formula 2 * CPUs + 1, your server maybe a single CPU machine, so set it to 3 is safe. |
|
any log to share? How to reproduce? By itself gunicorn don't use much ram or cpu. Also why are you preloading the app? |
|
is there a wat ri reproduce the issue? Gunicorn by itself doesn't use much cpu, it's more likely smth in your app is triggering the issue. |
|
bump. |
|
just set |
|
no answer since by the issuer closing the issue then. @hardiksanchawat you shouldn't use preload until you really need to use preload, which may be one of the reason. Also likesaid by @diwu1989 you can use the max-requests setting to prevent any memory leak. |
|
We have hosted our Web App (Angular CLI: 1.7.3, Node: 9.5.0, Django 2.0.3) on AWS free tier of Ubuntu(14.04.5 LTS) VM. We are using ELB (Elastic Load Balancers) with Nginx-1.4.6 and gunicorn-19.7.1 When we try to use access our App from a browser then suddenly gunicorn memory usage increases quite high: Normal: While using App: description "Gunicorn application server handling myproject" start on runlevel [2345] respawn exec ./env/bin/gunicorn --max-requests 1 --workers 3 --bind unix:/home/ubuntu/project/django-ng.sock config.wsgi:application We have already set max-requests as 1 and workers as 3. Could someone please tell what is going wrong? |
|
@satendrapratap I'm not sure anything is wrong. You might have to profile your app, but it's also possible that this is Python process on a 1GB machine and you have three workers. It may be normal. It's hard to say. |
|
Ok. Which profiler should we use to profile? Same code is running well with "python3 manage.py runserver" and we don't see any increase in memory used over a period of time or at once. Most probably there are no memory leaks otherwise there would have been increased memory consumption in this case also. How would profiling actually help? Are you saying that we need to optimize the code to make it consume less memory?
How much memory does a Python process takes at runtime? In our case, gunicorn takes as much as 500-600 MB with just 1 max-request and if we increase it to 2 then accessing app sometimes gives out of memory error. |
|
@satendrapratap profiling would help if there were a place you could reduce your own application's memory usage. It would also help to know if Gunicorn is using more memory than it should, or if there is a leak with Gunicorn that does not happen with runserver. |
|
I just did some quick tests of minimal applications at it seems to be as small as 20-30MiB for a worker, but for a real application it depends a lot on the application and the framework. I am interested in any comparison between Gunicorn and runserver you can provide. |
Note that Python has its own memory allocator and doesn't give previously allocated memory blocks back to OS so increased memory usage doesn't always mean that there is a memory leak. Do you see the same memory usage on all URLs or just a specific URL (/products/all for example)? I'd also suggest deploying a simple Django app (same database and Gunicorn configuration; just one model and a couple of views) to see whether Gunicorn is the cause of the high memory usage. |
We will definitely try profiling. Could you please suggest a profiler to use in such scenario?
We have 3 workers and while app isn't being used, its consuming around 96 MB : ubuntu@:~$ sudo python3 ps_mem.py 136.0 KiB + 9.0 KiB = 145.0 KiB acpid
|
|
how is memory consumption affected by increasing workers and max-requests? I am asking this because max-request = 1 is not going to be used in case of production and we will have to tune it and workers as well. so how much memory is usually required for small application deployed with Gunicorn and Nginx (20-30 rest services, only 4-5 out of them do some processing on the server while rest just takes data from the database and send back to the user, being hit by at least 10000 users simultaneously) ? |
You'd need to write a custom memory allocator by using Python C API but that's a non-trivial task and I don't think it's needed.
The table is a bit hard to read. I'm guessing the first column is PID, the second is owner of the process, but the rest is a bit hard to guess. Could you add column names too? In any case, the increase of memory usage from ~90 MB to ~500 MB is too much. I think we would already getting a lot of similar reports if we had introduced such a huge memory leak by now :) (Gunicorn 19.7.1 was released more than a year ago.) |
|
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND |
|
300MiB resident might not be unexpected, but I don't know. It is Python and an ORM, but I'm not a heavy Django user and haven't profiled these things before, myself. For 10,000 users, depending on the request rate, you may need more than a t2.micro. However, we'll do our best to help you troubleshoot. Having nginx in front for queuing requests to Gunicorn will certainly help. How much memory does the same application take when requesting the URLs against runserver? |
|
We are using t2.micro which has 1 GB of ram. Our setup is like below: nginx config:gunicorn config:
Using "python3 manage.py runserver" on local machine: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND First Access Web App to do some processing: Second Access Web App to do some processing: Third Access Web App to do some processing: |
|
Comparing these measurements to the earlier comment with Gunicorn memory stats, it seems like there is no problem with Gunicorn here. Am I misreading? |
|
@tilgovi I am currently working on similar problem and I wanted to add my findings in case it may help. So I assume something is caching things and do not leave them with reboot/restart. Is there any method I can monitor what is causing memory increase under gunicorn as I can only see %60 of memory is consumed by gunicorn.
|
|
@tilgovi we went through the details and looks like gunicorn and runserver results of memory consumption can't be compared because we configured gunicorn for max-requests 1 which will restart worker after every request so we won't really be able to tell whether memory increase or not for subsequent rest service calls. So set max-request as 100, workers as 3 and tried our experiment again. In idle situation gunicorn memory usage is fine: But we use the web app then worker consume more memory (maybe for python data structure etc)and after serving the request it does not return back (maybe because Python has its own memory allocator and doesn't give previously allocated memory blocks back to OS but would use this memory for further rest service requests) to normal memory consumption of 34868 : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND but when start using the web app then workers are chosen randomly and when the third worker is chosen to serve the request (after other 2 workers have already served and hold 270428 KB memory each) then our system tries to allocate memory which isn't available enough and worker crashes and restarts. So looks like we can debug the memory consumption properly only if we have enough memory available to us the app enough number of times without crashing workers and restarting. Then only we can identify whether memory used increases over a period of time (leak) or returns back to the first case where a worker is consuming memory for itself and for python allocation (which is normal situation). Worker crashes without completing the request because of less memory available: |
|
Forgive me, I'm still not fully understanding. It looks like Gunicorn uses ~300MiB. The only difference with max-requests is that you're more likely to have all three workers consuming ~300MiB each, rather than only the one that's currently handling a request. On the other hand, runserver is also using ~300MiB per process. It seems like your instance is not large enough to run three workers and it doesn't matter whether you use runserver or gunicorn. If you want to handle more concurrent requests, you'll need to use an async worker type rather than start multiple workers. Is there anything else I'm missing? |
|
I think you are absolutely right and nothing is missed. Actually, the problem was that Gunicron documentation suggests to user number of workers as 2*N + 1 and because we were having a single core so we used 3 workers. Now there is nothing mentioned about the manner these workers are used in. Looks like they are scheduled by OS and you never know which worker will handle incoming request. So initially what was happening was we had limited memory, out of which nginx and other system processes took some part then rest of the memory (~800 MB) was supposed to be used by workers. Because we configured for 3 workers so assuming that each worker takes 300 MB then all 3 together will consume 900 MB which isn't available so when the third worker (which you never know because its as per OS scheduling) is handling the request (after other 2 workers already handled and consuming total of 600 MB) then system doesn't have enough memory and worker crashes. This happened randomly so we configured max-request as 1 which was restarting worker after every request and freeing whatever memory was allocated for our python app. This way worker crash was overcome but system became slow and we thought its worker which suddenly takes huge memory and slows down the system and we even couldn't debug it as every time worker was restarted. To debug it more we had to have max-request big enough to serve more requests without restarting worker but that wasn't possible because if we allow more requests with 3 workers (each can take ~300 MB) in 1 GB system then worker was crashing and we were getting out of memory. But while we made the number of workers as 1 then the system is behaving more or less like runserver. So now we are not getting out of memory and system is fast also. So what we learned is that the general rule of creating 2*N+1 workers is just a vague logic as you need to consider memory as well. Suppose each gunicorn worker takes ‘W’ memory and total system memory is ‘T’ and having N=1 core. So as per the suggestion minimum number of workers = 2*1 + 1 = 3 Now suppose your application takes ‘A’ memory. So total memory required with only one worker handling all requests R = W*3 + A So as long as T is more enough than R, everything is fine but the problem comes when suppose Operating System schedule other workers to serve more requests then each worker consumes at least W+A memory. So actually system memory required for gunicorn with 3 workers should be more than (W+A)*3 to avoid random hangs, random no responses or random bad requests responses (for example nginx is used as reverse proxy then it will not get any response if gunicorn worker is crashing because of less memory and in turn nginx will respond with a Bad Request message) So we need to be careful while configuring gunicorn and consider both numbers of core and memory. I believe that Gunicorn documentation at http://docs.gunicorn.org/en/stable/settings.html |
|
Logic is not that vague ;) Having 2*cores + 1 allows good load balancing between workers. It's also true that each workers are isolated and will load the application independently and then consume some memory. If you want to reduce the size of workers on a limited machine you can use the |
|
also good point we should have a part about the memory in that design page. |
|
gunicorn definitely eats lots of mem than it should, recently I switch to uwsgi save losts of memory.. |
|
@tyan4g gunicorn itself don't use much memory, it doesn't buffer and has a pretty low memory footprint for a python application. What is using the ram is generally the application and its usage. All workers are isolated and by default the memory is not shared. uwsgi has a similar approach so it shouldn't impact the memory much. Note that using the gthread or any other async worker allows you to reduce the number of workers if needed. |
|
I also find the worker process increased suddently. Over time, we meet memory leak. Then I config gunicorn with max-request, then problem solved. I want to know the possible reason why i met memory leak when I start gunicorn without max_request params. The gunicorn version is 19.7.0 i used. |
|
@benoitc Can you help give me train of thought? |
|
@v-wiil you will probably need to investigate this on your own and open an new issue if you think there's a leak in Gunicorn. |
so its ok to have run 1.3 gb of ram utilization from 3 worker |
|
@vizvasrj the issue is more likerly in your app use a lot of memory. I bet if you start with the application preloaded the RAM will be at the same level at is in django. But you will loose the safety yoj get in using isolated workers. Let me know. |
|
thanks
…On Sat, 29 Jan 2022 at 17:24, Benoit Chesneau ***@***.***> wrote:
@vizvasrj <https://github.com/vizvasrj> the issue is more likerly in your
app use a lot of memory. I bet if you start with the application preloaded
the RAM will be at the same level at is in django. But you will loose the
safety yoj get in using isolated workers.
Let me know.
—
Reply to this email directly, view it on GitHub
<#1250 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AT2SBSVHA36URWH7XZIZD33UYPIO3ANCNFSM4CCQHKLQ>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
In django project, I am using Gunicorn as an application server. Last few days application is running smoothly for few hours after that application is hanged. Gunicorn is utilizing more memory, due to this CPU utilization crossed to 95% and application is hanged.
Here is my Gunicorn configuration while starting application.
gunicorn base.wsgi:application
--bind=0.0.0.0:8000
--pid=logs/project/gunicorn.log
--access-logfile=logs/project/access.log
--workers=7
--worker-class=gevent
--error-logfile=logs/project/error8.log
--timeout=4500
--preload
Please help me on this issue.
The text was updated successfully, but these errors were encountered: