Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Proxy doesn't change when using different username #308

Open
MrDebugger opened this issue Apr 30, 2024 · 1 comment
Open

[BUG] Proxy doesn't change when using different username #308

MrDebugger opened this issue Apr 30, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@MrDebugger
Copy link

Describe the bug:
The curl_cffi library does not switch proxies correctly when only the username in the proxy URL changes, while other components (host, port, and password) remain the same. This issue is observed when multiple proxies are used sequentially in a single session, resulting in all requests incorrectly using the first proxy's credentials.

To Reproduce:
Steps to reproduce the behavior:

Set up a Session with proxies that differ only in the username.
Execute requests to a target website (e.g., httpbin.org/ip) that returns the requester's IP.
Notice that the session fails to update the proxy after the initial set, using the first proxy for all requests.

from curl_cffi.requests import Session

proxies = {
    'Proxy1': 'http://username1:password@host:port',
    'Proxy2': 'http://username2:password@host:port',
}

s = Session()

for proxy_key, proxy_url in proxies.items():
    s.proxies = {
        'http': proxy_url,
        'https': proxy_url,
    }
    r = s.get('https://httpbin.org/ip')
    print(proxy_key, r.json())

Expected behavior:
Each request should reflect the IP of the corresponding proxy defined by its username. However, the actual output shows the IP of the first proxy for both requests, indicating that the proxy change was not recognized.

Versions:
OS: Ubuntu 20.04.4 LTS
curl_cffi version: 0.6.3
Python version: 3.10.12

Additional context:
The session used is synchronous.
The proxy credentials issue suggests a possible mishandling in the proxy management within the session, particularly with caching or parsing mechanisms.

@MrDebugger MrDebugger added the bug Something isn't working label Apr 30, 2024
@yifeikong
Copy link
Owner

yifeikong commented Apr 30, 2024

I'm afraid this is how libcurl works. See my comments here.

#210 (comment)

You can try adding Connection: close header or downgrading to HTTP/1.1, but it's not guaranteed to work.

IMHO, this is the correct behavior actually. Establishing new connection to the same address is quite inefficient, and these auto-rotating proxies abuse the behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants