Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClientConnectorError while using proxies in aiohttp requests with aiomultiprocess.Pool #102

Open
hen68 opened this issue Jun 26, 2021 · 0 comments

Comments

@hen68
Copy link

hen68 commented Jun 26, 2021

Code

from asyncio import gather
import asyncio
from aiomultiprocess import Pool, Worker
from aiohttp import ClientSession
import aiohttp



async def get(url, host, port, user, passwd):
    async with aiohttp.ClientSession() as session:
        proxy_auth = aiohttp.BasicAuth('user', 'passwd')
        async with session.get("http://python.org",
                               proxy="http://host:port",
                               proxy_auth=proxy_auth,
                               ) as resp:
            print(await resp.text())

async def main(host, port, user, passwd):
    urls=['http://python.org']
    async with Pool() as pool:
       async for result in pool.map(get, urls):
          print('done')
        

def start(host, port, user, passwd):
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
    asyncio.run(main(host, port, user, passwd))


if __name__ == "__main__":
    start(host, port, user, passwd)

Details

  • OS: Windows 10
  • Python version: 3.9.5
  • aiomultiprocess version: 0.9.0

Full traceback

Traceback (most recent call last):
  File "test.py", line 31, in <module>
    start()
  File "test.py", line 27, in start
    asyncio.run(main())
  File "c:\users\administrator\appdata\local\programs\python\python38\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "c:\users\administrator\appdata\local\programs\python\python38\lib\asyncio\base_events.py", line 616, in run_until_complete
    return future.result()
  File "test.py", line 21, in main
    async for result in pool.map(get, urls):
  File "C:\Users\Administrator\.virtualenvs\sizeer-QHiYSeeW\lib\site-packages\aiomultiprocess\pool.py", line 145, in results_generator
    yield (await self.pool.results([task_id]))[0]
  File "C:\Users\Administrator\.virtualenvs\sizeer-QHiYSeeW\lib\site-packages\aiomultiprocess\pool.py", line 308, in results
    raise ProxyException(tb)
aiomultiprocess.types.ProxyException: Traceback (most recent call last):
  File "C:\Users\Administrator\.virtualenvs\sizeer-QHiYSeeW\lib\site-packages\aiohttp\connector.py", line 969, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
  File "c:\users\administrator\appdata\local\programs\python\python38\lib\asyncio\base_events.py", line 1050, in create_connection
    transport, protocol = await self._create_connection_transport(
  File "c:\users\administrator\appdata\local\programs\python\python38\lib\asyncio\base_events.py", line 1080, in _create_connection_transport
    await waiter
  File "c:\users\administrator\appdata\local\programs\python\python38\lib\asyncio\proactor_events.py", line 395, in _loop_writing
    self._write_fut = self._loop._proactor.send(self._sock, data)
  File "c:\users\administrator\appdata\local\programs\python\python38\lib\asyncio\windows_events.py", line 529, in send
    self._register_with_iocp(conn)
  File "c:\users\administrator\appdata\local\programs\python\python38\lib\asyncio\windows_events.py", line 718, in _register_with_iocp
    _overlapped.CreateIoCompletionPort(obj.fileno(), self._iocp, 0, 0)
OSError: [WinError 87] The parameter is incorrect

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\Administrator\.virtualenvs\sizeer-QHiYSeeW\lib\site-packages\aiomultiprocess\pool.py", line 110, in run
    result = future.result()
  File "C:\Users\Administrator\Desktop\sizeer\test.py", line 12, in get
    async with session.get("http://python.org",
  File "C:\Users\Administrator\.virtualenvs\sizeer-QHiYSeeW\lib\site-packages\aiohttp\client.py", line 1117, in __aenter__
    self._resp = await self._coro
  File "C:\Users\Administrator\.virtualenvs\sizeer-QHiYSeeW\lib\site-packages\aiohttp\client.py", line 520, in _request
    conn = await self._connector.connect(
  File "C:\Users\Administrator\.virtualenvs\sizeer-QHiYSeeW\lib\site-packages\aiohttp\connector.py", line 535, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "C:\Users\Administrator\.virtualenvs\sizeer-QHiYSeeW\lib\site-packages\aiohttp\connector.py", line 890, in _create_connection
    _, proto = await self._create_proxy_connection(req, traces, timeout)
  File "C:\Users\Administrator\.virtualenvs\sizeer-QHiYSeeW\lib\site-packages\aiohttp\connector.py", line 1139, in _create_proxy_connection
    transport, proto = await self._wrap_create_connection(
  File "C:\Users\Administrator\.virtualenvs\sizeer-QHiYSeeW\lib\site-packages\aiohttp\connector.py", line 975, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host python.org:443 ssl:default [The parameter is incorrect]

Description

I get this error no matter if i use Worker or Pool. Without proxies it works fine, adding verify_ssl=False to request doesn't resolve the issue as well. What's more without using Pool or Worker, only with asyncio it works just fine both with proxies and without. However, with python 3.8.5 and macos it works, not sure why (downgrading python on windows didn't resolve the problem). I suppose it's an issue with aiomultiprocess. Thanks in advance for help!

@hen68 hen68 changed the title ClientConnectorError while using proxies in aiohttp requests ClientConnectorError while using proxies in aiohttp requests with aiomultiprocess.Pool Jun 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant