https:// does not work with Python 3.6 #3752

juokaz · 2016-12-07T20:32:44Z

When requesting a https:// url, request method falls into an infinite recursion.

The issue is this method in python/ssl.py.

    @options.setter
    def options(self, value):
        super(SSLContext, SSLContext).options.__set__(self, value)

The commit which introduced this line to python is this https://hg.python.org/cpython/rev/c32e9f9b00f7.

The actual error looks like this:

  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 501, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 594, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 835, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 311, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 264, in create_urllib3_context
    context.options |= options
  File "/usr/local/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/local/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/local/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 316 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object

Tested using python 3.6.0b4.

The text was updated successfully, but these errors were encountered:

Lukasa · 2016-12-07T20:52:07Z

This sounds like this is a bug in the standard library, actually: it seems like they introduced an infinite recursion when setting options directly. @tiran?

tiran · 2016-12-07T20:58:39Z

I can't reproduce the issue on my machine (Python from latest hg 3.6 branch)

$ ./python 
Python 3.6.0+ (3.6:c4f39b6f3176, Dec  7 2016, 21:55:50) 
[GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ctx = ssl.SSLContext(ssl.PROTOCOL_TLS)
>>> ctx.options |= 5
>>> class MySSLContext(ssl.SSLContext):
...     pass
... 
>>> ctx = MySSLContext(ssl.PROTOCOL_TLS)
>>> ctx.options |= 5

alexwlchan · 2016-12-07T21:00:17Z

I just had a try to reproduce it from the requests end; I can’t hit it on 3.6.0b4 or 3.6.0rc1:

$ python
Python 3.6.0b4 (v3.6.0b4:18496abdb3d5, Nov 21 2016, 20:44:47)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('https://github.com/kennethreitz/requests/issues/3752')
<Response [200]>

$ python
Python 3.6.0rc1 (v3.6.0rc1:29a273eee9a5, Dec  6 2016, 16:24:13)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('https://github.com/kennethreitz/requests/issues/3752')
<Response [200]>

Could it be something that’s platform or URL-specific?

(OS X 10.12.1, 16B2659, for my part)

tiran · 2016-12-07T21:02:16Z

Requests works with Python 3.6.0 from hg, too:

$ ./python -m venv venv
$ venv/bin/pip install requests
Collecting requests
  Using cached requests-2.12.3-py2.py3-none-any.whl
Installing collected packages: requests
Successfully installed requests-2.12.3
$ venv/bin/python
Python 3.6.0+ (3.6:c4f39b6f3176, Dec  7 2016, 21:55:50) 
[GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('https://www.python.org')
<Response [200]>

Lukasa · 2016-12-07T21:02:53Z

Ok, so right now this looks like a "can't repro" situation. Might be worth looking at your install @juokaz, seems to be misbehaving a bit.

juokaz · 2016-12-07T21:14:03Z

After further debugging, I was able to reproduce it as a gevent issue (which was the context I was using this in).

I'm guessing I'm going to have to take it to gevent folks to have a look? Thanks for looking into this, everyone.

root@8042e3f57981:/app# python
Python 3.6.0b4 (default, Nov 23 2016, 21:34:29)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gevent.monkey
>>> gevent.monkey.patch_all()
>>>
>>> from requests.packages.urllib3.util.ssl_ import create_urllib3_context
>>> create_urllib3_context()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 268, in create_urllib3_context
    context.options |= options
  File "/usr/local/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/local/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  File "/usr/local/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
  [Previous line repeated 329 more times]
RecursionError: maximum recursion depth exceeded while calling a Python object
>>>

Lukasa · 2016-12-07T21:14:42Z

Yup, this is a gevent issue. I'd assume gevent doesn't yet work with Python 3.6.

tiran · 2016-12-07T21:14:48Z

The super() call looks weird but it's correct. It's the generic way to fetch the attribute descriptor of the parent class. It's a necessary evil to change a property in a subclass. Ask Google for "David Beazley subclass property" and it will show you the same, https://www.safaribooksonline.com/library/view/python-cookbook-3rd/9781449357337/ch08s08.html

>>> from ssl import SSLContext
>>> super(SSLContext, SSLContext).options
<attribute 'options' of '_ssl._SSLContext' objects>
>>> import _ssl
>>> super(SSLContext, SSLContext).options is _ssl._SSLContext.options
True

The call super(SSLContext, SSLContext).options.__set__(self, value) is the same as setting options on an instance of the parent class.

pirate · 2017-01-19T02:10:34Z

Here's the reference to the gevent ticket for the same issue: gevent/gevent#903
Gevent has now been bumped to >1.2 by default. If you're still having this issue, it can be fixed with a simple pip install --upgrade gevent.

AeroNotix · 2017-04-04T10:52:45Z

@pirate that doesn't fix the issue for me. Any other ideas? The other thread is locked.

sigmavirus24 · 2017-04-04T14:32:36Z

@AeroNotix please don't use this issue or this project's tracker to discuss gevent problems.

AeroNotix · 2017-04-04T14:45:09Z

@pirate can you unlock that ticket, then? I'm still having the same issue on python 3.6 with upgraded versions of everything.

pirate · 2017-04-04T17:49:52Z

Nope @AeroNotix , I'm not a contributor on either project, perhaps you can email/irc one of the people in the other thread directly?

rossengeorgiev · 2017-04-17T21:45:19Z

@Pirata This issue still exists with python 3.6.1, gevent 1.2.1, requests 2.13.0. Steps to reproduce:

import requests
import gevent.monkey
gevent.monkey.patch_ssl()
requests.get("https://google.com")

If I import requests after the patch, everything works. It used to be the case that it won't matter and I'm wondering if something can be changed to keep the old behavior. What do you think @Lukasa ?

Lukasa · 2017-04-17T22:06:21Z

Requests has some problems with gevent and monkey patching because it performs feature detection on the select module at import time. A future release will bring it up to date with a newer urllib3 that contains fixes for this problem, but in the meantime gevent's monkey patch should be applied before importing requests.

But in questions is psf/requests#3752 Workaround includes two new entry points, talisker.gunicorn.gevent and talisker.gunicorn.eventlet, that ensure monkeypatching is done before requests is imported. In the procss, have refactored all entrypoint functions to be in __init__.py. This moves it all into one place, and helps avoiding early imports.

* Workaround async requests bug on py36 But in questions is psf/requests#3752 Workaround includes two new entry points, talisker.gunicorn.gevent and talisker.gunicorn.eventlet, that ensure monkeypatching is done before requests is imported. In the procss, have refactored all entrypoint functions to be in __init__.py. This moves it all into one place, and helps avoiding early imports. * Add docs for new workarounds * increase timeout in py36 tests, hopefully temporarily

ikonst · 2018-07-16T18:45:50Z

@Lukasa are you referring to this release?

juokaz changed the title ~~https:// not working with Python 3.6~~ https:// does not work with Python 3.6 Dec 7, 2016

Lukasa closed this as completed Dec 7, 2016

bloodearnest mentioned this issue May 31, 2017

Requests error during ssl with eventlet canonical-ols/talisker#98

Closed

bloodearnest mentioned this issue May 31, 2017

Workarounds for py36/async/requests bug canonical-ols/talisker#99

Merged

jobi mentioned this issue Sep 15, 2017

Infinite recursion error when testing https sites locustio/locust#655

Closed

jschneier mentioned this issue Aug 24, 2018

maximum recursion depth exceeded. jschneier/django-storages#560

Closed

bloodearnest mentioned this issue Nov 12, 2019

Should Talisker define a "gevent" extra? canonical-ols/talisker#491

Closed

bblommers mentioned this issue May 12, 2020

AWS CodeBuild, DynamoDB & mock_dynamodb2 getmoto/moto#2990

Closed

chlowell mentioned this issue Jun 15, 2021

SDK Usage in unittests end up in azure.core.exceptions.ClientAuthenticationError:uthentication failed: maximum recursion depth exceeded while calling a Python object Azure/azure-sdk-for-python#19264

Closed

github-actions bot locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

https:// does not work with Python 3.6 #3752

https:// does not work with Python 3.6 #3752

juokaz commented Dec 7, 2016 •

edited

Lukasa commented Dec 7, 2016

tiran commented Dec 7, 2016

alexwlchan commented Dec 7, 2016 •

edited

tiran commented Dec 7, 2016

Lukasa commented Dec 7, 2016

juokaz commented Dec 7, 2016

Lukasa commented Dec 7, 2016

tiran commented Dec 7, 2016

pirate commented Jan 19, 2017 •

edited

AeroNotix commented Apr 4, 2017

sigmavirus24 commented Apr 4, 2017

AeroNotix commented Apr 4, 2017

pirate commented Apr 4, 2017

rossengeorgiev commented Apr 17, 2017

Lukasa commented Apr 17, 2017

ikonst commented Jul 16, 2018

https:// does not work with Python 3.6 #3752

https:// does not work with Python 3.6 #3752

Comments

juokaz commented Dec 7, 2016 • edited

Lukasa commented Dec 7, 2016

tiran commented Dec 7, 2016

alexwlchan commented Dec 7, 2016 • edited

tiran commented Dec 7, 2016

Lukasa commented Dec 7, 2016

juokaz commented Dec 7, 2016

Lukasa commented Dec 7, 2016

tiran commented Dec 7, 2016

pirate commented Jan 19, 2017 • edited

AeroNotix commented Apr 4, 2017

sigmavirus24 commented Apr 4, 2017

AeroNotix commented Apr 4, 2017

pirate commented Apr 4, 2017

rossengeorgiev commented Apr 17, 2017

Lukasa commented Apr 17, 2017

ikonst commented Jul 16, 2018

juokaz commented Dec 7, 2016 •

edited

alexwlchan commented Dec 7, 2016 •

edited

pirate commented Jan 19, 2017 •

edited