gevent + requests blocks when using socks

[xsren]

I use python(2.7.6),gevnet(1.1.2),requests(2.11.1) to make http requests concurrent, and it works good. But when I add socks proxy to requests, it blocks.

This is my code:

import time
import requests
import logging
import click

import gevent
from gevent import monkey
monkey.patch_all()

FORMAT = '%(asctime)-15s %(message)s'
logging.basicConfig(format=FORMAT)
logger = logging.getLogger('test')

#socks proxy
user = MY_SOCKS_PROXY_USERNAME
password = MY_SOCKS_PROXY_PASSWORD
host = MY_SOCKS_PROXY_HOST
port = MY_SOCKS_PROXY_PORT
proxies = {
        'http': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
        'https': 'socks5://{0}:{1}@{2}:{3}'.format(user, password, host, port),
}
url = 'https://www.youtube.com/user/NBA'


def fetch_url(i,with_proxy):
    while True:
        logger.warning('thread %s fetch url'%i)
        try:
            if with_proxy:
                res = requests.get(url,proxies=proxies, timeout=5)
            else:
                res = requests.get(url, timeout=5)
        except Exception as e:
            logger.error(str(e))
            continue

        logger.warning(res.status_code)

def do_other_thing():
    while True:
        logger.warning('do other thing...')
        time.sleep(1)

@click.command()
@click.option('--with_proxy/--without_proxy',help='if use proxy', default=True)
def run(with_proxy):
    if with_proxy:
        logger.warning('with proxy......')
    else:
        logger.warning('without proxy......')
    ts = []
    ts.append(gevent.spawn(do_other_thing))
    for i in xrange(3):
        ts.append(gevent.spawn(fetch_url,i,with_proxy))
    gevent.joinall(ts)

if __name__=='__main__':
    run()

These pictures show the result.

run with proxy
run without proxy

With proxy, do_other_thing will blocks before fetch_url done.
Without proxy， it works good. (timeout error occurs, because of the GFW)

Can anyone help me solving this problem? Thanks very much!

[Lukasa]

My best guess is that you are monkeypatching too late. When using gevent you must monkeypatch first, before importing anything else. Doing it later runs the risk of breaking anything that has done a from import from the things that gevent monkeypatches.

Can you try moving your import and monkeypatch to the very top of your file before doing anything else?

[xsren]

Yeah, ' moving your import and monkeypatch to the very top of your file ' solved my problem! Thank you very much!

This suggestion also solves my project's problem.
I hava 3 files called A,B,C, and run sequence is A->B->C. I use gevent in B, so I add monkeypatch in B, but i blocks with requests+socks. Now I add monkeypatch at the very top of A, it works great now! Hope this will help others!

Thank you again!

[Lukasa]

Happy to help!