[BUG] Proxy rotation is not possible when using a session

[ZentixUA]

Describe the bug
When using curl-cffi, using a proxy through Proxy Gateway for rotation on each request becomes impossible (the IP becomes static for some reason, despite the fact that static sessions are disabled in the proxy provider's settings).

To Reproduce

1. Install aiohttp using pip install aiohttp
2. Install httpx using pip install httpx
3. Install curl-cffi using pip install curl-cffi
4. Paste this example:

import asyncio
import sys

from aiohttp import ClientSession
from curl_cffi.requests import AsyncSession
from httpx import AsyncClient

PROXY_URL = 'YOUR_PROXY_GATEWAY_URL'
REPETITION_COUNT = 3


async def test_aiohttp():
    print('----- aiohttp -----')
    async with ClientSession() as session:
        for i in range(REPETITION_COUNT):
            async with session.get('http://httpbin.org/get', proxy=PROXY_URL) as response:
                response_json = await response.json()
            origin = response_json['origin']
            print(f'Request #{i}, origin: {origin}')


async def test_httpx():
    print('----- httpx -----')
    async with AsyncClient(proxy=PROXY_URL) as session:
        for i in range(REPETITION_COUNT):
            response = await session.get('http://httpbin.org/get')
            origin = response.json()['origin']
            print(f'Request #{i}, origin: {origin}')


async def test_curl_cffi():
    print('----- curl-cffi -----')
    async with AsyncSession(proxy=PROXY_URL) as session:
        for i in range(REPETITION_COUNT):
            response = await session.get('http://httpbin.org/get')
            origin = response.json()['origin']
            print(f'Request #{i}, origin: {origin}')


if __name__ == '__main__':
    if sys.platform == 'win32':
        asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
    asyncio.run(test_aiohttp())
    asyncio.run(test_httpx())
    asyncio.run(test_curl_cffi())

5. Replace PROXY_URL with the URL of your proxy gateway
6. Run this code and look at the output in the console :)

Expected behavior
The IP changes with each request, similar to how it happens in aiohttp or httpx.

Versions

• OS: Windows 11
• curl_cffi version: 0.6.2
• pip freeze dump:

aiohttp==3.9.4
aiosignal==1.3.1
anyio==4.3.0
async-timeout==4.0.3
attrs==23.2.0
certifi==2024.2.2
cffi==1.16.0
curl_cffi==0.6.2
exceptiongroup==1.2.0
frozenlist==1.4.1
h11==0.14.0
httpcore==1.0.5
httpx==0.27.0
idna==3.7
multidict==6.0.5
pycparser==2.22
sniffio==1.3.1
typing_extensions==4.11.0
yarl==1.9.4

Additional context

• Which session are you using? Async.
• If using async session, which loop implementation are you using? Asyncio.

[yifeikong]

Things are complicated when you use so-called tunnel proxies where the exit node changes but proxy server url does not change. When the proxy url stays the same, it's reasonable that the client reuses the old connections, actually I think the way libcurl handles it is the correct way.

A few things you can try to see if it helps:

• Turn off http/2, requests.get(url, http_version=CurlHttpVersion.V1_1)
• Use the proxy when requesting, session.get(url, proxy=...)

[ZentixUA]

The solutions you provided did not work (at least in my case). However, I found two solutions myself.

According to this documentation, the Connection header can be set to close to ensure that the connection will close after processing the request. HTTP 1.0 has close as the default, so here are two solutions:

1. headers={'Connection': 'close'}
2. http_version=CurlHttpVersion.V1_0

[yifeikong]

Thanks for posting your solution here. I would not recommend http/1.0 as it has been deprecated for a very long time. Adding connection: close is the right way to go if it solves your problem.