fails on older TLS stacks with OpenSSL 1.1.1pre9

[anarcat]

The upcoming adoption of OpenSSL 1.1 (or more specifically, 1.1.1) by Linux distributions might cause problems for requests when checking old websites.

This was first reported in the Debian bugtracker as bug #907807, originally against the linkchecker program (and forwarded upstream as linkchecker/linkchecker#188), but it was found that the requests library directly suffers from this problem as well.

This issue will probably block requests from being released with buster, the current "testing" release and upcoming (mid 2019?) stable release unless it is somewhat fixed. That assertion was incorrect: the bug is currently marked with a "normal" severity which is not blocking release.

The tested sites were:

• https://get.adobe.com/
• https://caniuse.com
• https://www.nada.kth.se/~snilsson/publications/IP-address-lookup-using-LC-tries/

All sites load correctly in Firefox 60.1.0 and although I haven't tested that with OpenSSL 1.1.1, I doubt it would be affected as Firefox (and Chromium) have their own TLS library (NSS). Also note that urllib3 seems to have no problem loading those sites itself:

>>> import urllib3
>>> http = urllib3.PoolManager()
>>> r = http.request('GET', 'https://get.adobe.com/')
>>> 

Expected Result

All sites should load correctly, and do load correctly in Debian buster (which still has OpenSSL 1.1.0):

$ python
Python 2.7.15+ (default, Aug 31 2018, 11:56:52) 
[GCC 8.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get('https://get.adobe.com')
<Response [200]>

Actual Result

(unstable-amd64-sbuild)anarcat@curie:/$ python
Python 2.7.15+ (default, Aug 31 2018, 11:56:52) 
[GCC 8.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
 >>> requests.get('https://get.adobe.com')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
     return request('get', url, params=params, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='get.adobe.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: WRONG_SIGNATURE_TYPE] wrong signature type (_ssl.c:726)'),))
 >>> requests.get('https://www.nada.kth.se')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
     return request('get', url, params=params, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='www.nada.kth.se', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:726)'),))
 >>> requests.get('https://caniuse.com')
 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 72, in get
     return request('get', url, params=params, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/api.py", line 58, in request
     return session.request(method=method, url=url, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
     resp = self.send(prep, **send_kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
     r = adapter.send(request, **kwargs)
   File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 506, in send
     raise SSLError(e, request=request)
 requests.exceptions.SSLError: HTTPSConnectionPool(host='caniuse.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: VERSION_TOO_LOW] version too low (_ssl.c:726)'),))

Reproduction Steps

Run a Debian unstable machine to get the latest 1.1.1~~pre9 package. This can be done with Docker with:

docker run -it debian:unstable

If you do not have access to such an environment, the latest OpenSSL code can be found in their project page.

import requests

for url in ('https://get.adobe.com/', 'https://caniuse.com',
'https://www.nada.kth.se/~snilsson/publications/IP-address-lookup-using-LC-tries/'):
    requests.get(url)

System Information

$ python -m requests.help

(unstable-amd64-sbuild)anarcat@curie:/$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  }, 
  "cryptography": {
    "version": ""
  }, 
  "idna": {
    "version": "2.6"
  }, 
  "implementation": {
    "name": "CPython", 
    "version": "2.7.15+"
  }, 
  "platform": {
    "release": "4.17.0-3-amd64", 
    "system": "Linux"
  }, 
  "pyOpenSSL": {
    "openssl_version": "", 
    "version": null
  }, 
  "requests": {
    "version": "2.18.4"
  }, 
  "system_ssl": {
    "version": "10101009"
  }, 
  "urllib3": {
    "version": "1.22"
  }, 
  "using_pyopenssl": false
}

I am aware that we are one minor version behind upstream in Debian, and this is being worked on. But I have reviewed the requests changelog and there didn't seem to be any change relevant to this. I am therefore going under the assertion that current versions also suffer from the same problem.

Also note that this might be a regression in OpenSSL itself. I am filing this here to see what your opinion is regarding this issue and whether it is something that belongs in requests or the upstream cryptographic library(ies).

[kroeckx]

For the adobe problem I've opened: openssl/openssl#7126

[kroeckx]

The caniuse problem is: Fyrd/caniuse#4198

It's caused by Debian setting a minimum TLS version of 1.2. They really should add at least TLS 1.2 support.

[kroeckx]

For the SNI problem, see: https://wiki.openssl.org/index.php/TLS1.3#Server_Name_Indication

You will get that problem for anything hosted by Google.

[kroeckx]

So some of those issues are caused by Debian increasing the security level of openssl by default. The proper fix is to get the websites fixed. But it's possible to lower those security settings as a work around. Please don't override the defaults by default.

The SNI problem is something you really should fix. I don't know which part doesn't set it.

[ziima]

Hi, could you provide the workarounds? I have tried to lower the security connection to avoid DH_KEY_TOO_SMALL, but I couldn't find any solution which would work for me (Debian testing, requests installed from debian package). Specifically for https://www.ceskatelevize.cz/. It's not very likely they will fix their security any time soon.

[kroeckx]

See /usr/share/doc/libssl1.1/NEWS.Debian.gz

[ziima]

I've seen that, but I don't want to lower the security for the whole system, just for specific script. I've also tried https://stackoverflow.com/q/38015537/2440346, but change of urllib3.util.ssl_.DEFAULT_CIPHERS doesn't work either.

[kroeckx]

If you want to lower the security setting in an other file than /etc/ssl/openssl.cfg you need to use DEFAULT@SECLEVEL=1

[ziima]

I've tried number of approaches, but none work, see

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context

# Attempt 1:
# requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = '@SECLEVEL=1'
# Attempt 2:
# requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'DEFAULT@SECLEVEL=1'
# Attempt 3:
# requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'ALL'
requests.get('https://www.ceskatelevize.cz/')

# Attempt 4:
# CIPHERS = '@SECLEVEL=1'
# Attempt 5:
# CIPHERS = 'DEFAULT@SECLEVEL=1'
# Attempt 6:
CIPHERS = 'ALL'
class CustomAdapter(HTTPAdapter):
    """
    A TransportAdapter that re-enables 3DES support in Requests.
    """
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        return super(CustomAdapter, self).init_poolmanager(*args, **kwargs)

    def proxy_manager_for(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        return super(CustomAdapter, self).proxy_manager_for(*args, **kwargs)

session = requests.Session()
session.mount('https://www.ceskatelevize.cz', CustomAdapter())
session.get('https://www.ceskatelevize.cz/')

[Matir]

The following works for me:

import requests
from requests import adapters
import ssl
from urllib3 import poolmanager


class TLSAdapter(adapters.HTTPAdapter):

    def init_poolmanager(self, connections, maxsize, block=False):
        """Create and initialize the urllib3 PoolManager."""
        ctx = ssl.create_default_context()
        ctx.set_ciphers('DEFAULT@SECLEVEL=1')
        self.poolmanager = poolmanager.PoolManager(
                num_pools=connections,
                maxsize=maxsize,
                block=block,
                ssl_version=ssl.PROTOCOL_TLS,
                ssl_context=ctx)

session = requests.session()
session.mount('https://', TLSAdapter())
session.get(TARGET)

[codefather-labs]

The following works for me:

import requests
from requests import adapters
import ssl
from urllib3 import poolmanager


class TLSAdapter(adapters.HTTPAdapter):

    def init_poolmanager(self, connections, maxsize, block=False):
        """Create and initialize the urllib3 PoolManager."""
        ctx = ssl.create_default_context()
        ctx.set_ciphers('DEFAULT@SECLEVEL=1')
        self.poolmanager = poolmanager.PoolManager(
                num_pools=connections,
                maxsize=maxsize,
                block=block,
                ssl_version=ssl.PROTOCOL_TLS,
                ssl_context=ctx)

session = requests.session()
session.mount('https://', TLSAdapter())
session.get(TARGET)

Thanks!
Its helpfuly for my SSLError [SSL: DH_KEY_TOO_SMALL]

[riccardotacconi]

@codefather-labs thanks I had the same issue, very odd

[sethmlarson]

This very much seems like a server-side/OpenSSL config problem instead of anything Requests can "work-around". How OpenSSL is configured on a platform can't be controlled by Requests maintainers. Going to close this as not a defect in Requests.