Too many open files

[daleharvey]

I am building a basic load generator and started running into file descriptor limits, I havent seen any documentation pertaining to how to release resources, so either I am doing it wrong and the docs need updated, or requests is leaking file descriptors somewhere (without support for keepalive I am slightly confused at why any files would be left open at all)

[kennethreitz]

Where you using requests.async?

[daleharvey]

nope, all requests were reasonably plain requests.get / requests.post, I am still seeing a few in there

$ lsof | grep localhost | wc -l
110

all but 4/5 of them are of the format

Python    82117 daleharvey  123u    IPv4 0xffffff800da304e0       0t0      TCP localhost:61488->localhost:http (CLOSE_WAIT)

[kennethreitz]

I'm a bit baffled by this, to be honest.

[daleharvey]

Hah ill take another shot at reproducing it reliably, if I cant ill close

[kennethreitz]

I've seen this happening to me, but only when I'm using the async module w/ 200+ simultaneous connections.

[mike-plivo]

Hi,
I got exactly the same problem using requests and monkey patching with gevent : some connections staying in CLOSE_WAIT .
Maybe a problem with gevent so .

[juanriaza]

It may be problem of ulimit -n. Try with a higher value.

[mike-plivo]

"Too many open files" is the result of the bug caused by sockets staying in CLOSE_WAIT .
So ulimit won't fix just make a workaround .

[kennethreitz]

@Tamiel how do we fix this?

[mike-plivo]

I will do more tests asap and try to fix .

[kennethreitz]

I've looked into it, and seems to be a problem with all libraries using httplib.HTTPSConnection.

[mike-plivo]

Posted an example here :

https://gist.github.com/1512329

[acdha]

I just encountered a very similar error using an async pool with only HTTP connections - I'm still investigating but passing a pool size to async.map makes the error reproduce quickly.

[bevenky]

Any fixes to this? This makes Requests unusable with gevent..

[kennethreitz]

It's all about the CLOSE_WAITs. Just have to close them. I'm not sure why they're still open though.

[bevenky]

Is it a urllib3 issue? Having to close these by ourselves isnt a great idea i feel.

[kennethreitz]

It's more of a general issue. We can keep the conversation here.

[bevenky]

Ok just to give you a perspective, we are trying to move from httplib2 to requests, and we dont see this issue with httplib2. So its not a general issue for sure.

[kennethreitz]

By general i mean that it's a very serious issue that effects everyone involved.

[bevenky]

so how do we solve this? we really want to use requests + slumber moving forward

[kennethreitz]

I'd love to know the answer to that.

[acdha]

The leak appears to be due to the internal redirect handling, which causes new requests to be generated before the pending responses have been consumed. In testing acdha/requests@730c0e2 has an under-satisfying but effective fix simply by forcing each response to be consumed before continuing.

This required changes in two places which makes me want to refactor the interface slightly but I'm out of time to continue currently.

[acdha]

#399 has a fix which works well in my async load generator (https://github.com/acdha/webtoolbox/blob/master/bin/http_bench.py) with thousands of requests and a low fd ulimit

[bborowin]

I have run into the same issue when using async -- kludged a workaround by chunking requests and deleting responses / calling gc.collect

[Cabalist]

I believe I was running into this today connecting to a licensed server that only allows 5 connections.

Using async I could only GET 4 things before it paused for 60 seconds.

Using the normal GET with consumption I could fetch about 150 things serially in under 40 seconds.

Haven't made my kludge yet since I saw this issue.

[dalanmiller]

Just got this error while using ipython and got this message. This is just making each request one at a time, but I think I got something similar when using async.

ERROR: Internal Python error in the inspect module.
Below is the traceback from this internal error.
Traceback (most recent call last):
    File "/Library/Python/2.7/site-packages/IPython/core/ultratb.py", line 756, in structured_traceback
    File "/Library/Python/2.7/site-packages/IPython/core/ultratb.py", line 242, in _fixed_getinnerframes
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 1035, in getinnerframes
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 995, in getframeinfo
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 456, in getsourcefile
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 485, in getmodule
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 469, in getabsfile
    File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 347, in abspath
OSError: [Errno 24] Too many open files

Unfortunately, your original traceback can not be constructed.

Oddly, I think when using just the normal Python interpreter I get a "Max Retries Error" but I think that is another issue with me doing requests on all the same domain, but not sure.

[Lukasa]

@polvoazul There's no way this is the same issue, which was originally reported in 2011, so I don't think reopening is correct. However, if you're running the current release of requests (2.4.3) and can reproduce the problem, opening a new issue would be correct.

[mygoda]

@Lukasa i need you help 。 i use eventlet + requests，that always create so many sock that can't identify protocol 。 my requests is 2.4.3, is eventlet + requests cause this problem?

[Lukasa]

I'm sorry @mygoda, but it's impossible to know. If you aren't constraining the number of requests that can be outstanding at any one time then it's certainly possible, but that's an architectural problem outside the remit of requests.

[mygoda]

@Lukasa thank you。 i think my issue is similar with this。 my project is pyvmomi. that connection is long-connection. i always confused why can hold so many can't identify protocol sock

[1a1a11a]

Having the same problem now, running 120 threads, cause 100000+ opened files, any solution right now?

[kennethreitz]

@mygoda you use awesome periods。

[Lukasa]

@1a1a11a What files do you have open? That'd be a useful first step to understanding this problem.

[sigmavirus24]

@1a1a11a what version of requests are you using? What version of python? What operating system? Can we get any information?

[1a1a11a]

I am using request 2.9.1, python 3.4, ubuntu 14.04, basically I am writing a crawler using 30 threads with proxies to crawl some website. Currently I have adjusted the file limit per process to 655350, otherwise it will report error.

[miloharper]

I am still receiving the error "Failed to establish a new connection: [Errno 24] Too many open files" from requests.packages.urllib3.connection.VerifiedHTTPSConnection." I'm using Python 3.4, requests 2.11.1 and requests-futures 0.9.7. I appreciate requests-futures is a separate library, but it seems like the error is coming from requests. I'm attempting to make 180k asynchronous requests over SSL. I've divided those requests into segments of 1000, so I only move onto the next 1000 once all the future objects have been resolved. I'm running Ubuntu 16.04.2 and my default open files limit is 1024. It would be good to understand the underlying reason for this error. Does the requests library create an open file for each individual request? And if so, why? Is this a SSL certificate file? And does the requests library automatically close those open files when the future object is resolved?

[Lukasa]

Requests opens many files. Some of those files are opened for certificates, but they are opened by OpenSSL and not by Requests, so those aren't maintained. Additionally, Requests will also open, if needed, the .netrc file, the hosts file, and many others.

You will be best served by using a tool like strace to work out which files are opened. There is a strict list of system calls that lead to file descriptors being allocated, so you should reasonably swiftly be able to enumerate them. That will also let you know whether there is a problem or not. But, yes, I'd expect that if you're actively making 1000 connections over HTTPS then at peak load we could easily use over 1000 FDs.

[tkaria]

I struggled with this issue as well and found that using opensnoop on OS X worked great to let me see what was happening if anyone runs in to the same issues.

[nyurik]

I'm also frequently seeing this error when repeatedly calling requests.post(url, data=data) to an HTTP (not HTTPS) server. Running on Ubuntu 16.04.3, Python 3.5.2, requests 2.9.1

[Lukasa]

What is data?

[nyurik]

A few hundred kb text

[Lukasa]

Not a file object?

[nyurik]

No, I form a large query in memory.

[Lukasa]

Are you running this code in multiple threads?

[nyurik]

No, single thread, posting to localhost

[Lukasa]

It seems almost impossible for us to be leaking that many FDs then: we should be repeatedly using the same TCP connection or aggressively closing it. Want to check what your server is up to?

[iraykhel]

I am having this problem. Python 2.7, requests 2.18.4, urllib3 1.22.
Running multi-threaded code (not multi-processed). Connecting to at most 6 URLs at one time, manually creating and closing a new session for each one.

[mcobzarenco]

I'm having the same problem on Python 3.5, requests==2.18.4

[arryon]

@mcobzarenco are you sure you are (implicitly) closing the underlying connection of the response? Simply returning the response will not close the connection. When reading response.content the data is actually read and after that the socket will not stay in CLOSE_WAIT.