Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

returns.json decorator results in "Too many open files" error with AiohttpClient #232

Open
selimt opened this issue Aug 28, 2021 · 0 comments

Comments

@selimt
Copy link

selimt commented Aug 28, 2021

If I use the returns.json decorator with a consumer method that uses an AiohttpClient consumer then I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/tmp/doNotRemove/projects/catalog-client/catalog/utils.py", line 186, in async_create_namespace
    results = loop.run_until_complete(asyncio.gather(*futures))
  File "/usr/lib64/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/var/tmp/doNotRemove/projects/catalog-client/catalog/api.py", line 1264, in create_namespace_async
    return await self.__create_namespace_async(namespace)
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/clients/io/asyncio_strategy.py", line 38, in execute
    response = yield from executable.execute()
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/clients/io/asyncio_strategy.py", line 22, in invoke
    response = yield from callback.on_success(response)
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/clients/io/asyncio_strategy.py", line 20, in invoke
    response = yield from callback.on_failure(type(error), error, tb)
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/clients/io/execution.py", line 108, in on_failure
    return self._io.fail(exc_type, exc_val, exc_tb)
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/clients/io/interfaces.py", line 303, in fail
    compat.reraise(exc_type, exc_val, exc_tb)
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/six.py", line 719, in reraise
    raise value
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/clients/io/asyncio_strategy.py", line 17, in invoke
    response = yield from func(*args, **kwargs)
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/clients/aiohttp_.py", line 29, in new_callback
    response = yield from coroutine_callback(response)
  File "/usr/lib64/python3.7/asyncio/coroutines.py", line 120, in coro
    res = func(*args, **kw)
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/hooks.py", line 20, in wrapper
    return hook(*args, **kwargs)
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/returns.py", line 39, in __call__
    return self._strategy(*args, **kwargs)
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/returns.py", line 87, in __call__
    content = response.json()
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/clients/aiohttp_.py", line 155, in __call__
    with AsyncioExecutor() as executor:
  File "/var/tmp/doNotRemove/venvs/catalog_client_python3.7/lib64/python3.7/site-packages/uplink/clients/aiohttp_.py", line 184, in __init__
    self._loop = asyncio.new_event_loop()
  File "/usr/lib64/python3.7/asyncio/events.py", line 762, in new_event_loop
    return get_event_loop_policy().new_event_loop()
  File "/usr/lib64/python3.7/asyncio/events.py", line 660, in new_event_loop
    return self._loop_factory()
  File "/usr/lib64/python3.7/asyncio/unix_events.py", line 51, in __init__
    super().__init__(selector)
  File "/usr/lib64/python3.7/asyncio/selector_events.py", line 60, in __init__
    self._make_self_pipe()
  File "/usr/lib64/python3.7/asyncio/selector_events.py", line 107, in _make_self_pipe
    self._ssock, self._csock = socket.socketpair()
  File "/usr/lib64/python3.7/socket.py", line 491, in socketpair
    a, b = _socket.socketpair(family, type, proto)
OSError: [Errno 24] Too many open files

To Reproduce
Create an AiohttpClient with a POST consumer method

create about 1000 futures like this:

    loop = asyncio.get_event_loop()
    for i in range(count):
         futures.append(consumer_instance.consumer_method())
    results = loop.run_until_complete(asyncio.gather(*futures))

Expected behavior
I should be able to pass as many as futures as needed for it to be processed through asyncio

Additional context
As a workaround I removed that decorator and I am now retrieving the json results as a separate gather step like this:

    futures = []
    for r in results:
        futures.append(r.json())
    jsons = loop.run_until_complete(asyncio.gather(*futures))

I am running with uplink 0.9.4 aiohttp 3.7.4.post0 and Python 3.7.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant