HTTPie ignores system certificates

[ghost]

HTTPie ignores system certificates

http --debug -j https://example_using_my_ca.com

HTTPie 0.9.3
HTTPie data: /home/lukas/.httpie
Requests 2.10.0
Pygments 1.6
Python 3.4.3 (default, Oct 14 2015, 20:28:29) 
[GCC 4.8.4] linux

>>> requests.request(**{'allow_redirects': False,
 'auth': None,
 'cert': None,
 'data': '',
 'files': DataDict(),
 'headers': {'Accept': b'application/json',
             'Content-Type': b'application/json',
             'User-Agent': b'HTTPie/0.9.3'},
 'method': 'get',
 'params': ParamsDict(),
 'proxies': {},
 'stream': True,
 'timeout': 30,
 'url': 'https://example_using_my_ca.com',
 'verify': True})

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 578, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 351, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 814, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connection.py", line 289, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/util/ssl_.py", line 308, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.4/ssl.py", line 365, in wrap_socket
    _context=self)
  File "/usr/lib/python3.4/ssl.py", line 601, in __init__
    self.do_handshake()
  File "/usr/lib/python3.4/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/requests/adapters.py", line 403, in send
    timeout=timeout
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/urllib3/connectionpool.py", line 604, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/http", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.4/dist-packages/httpie/core.py", line 115, in main
    response = get_response(args, config_dir=env.config.directory)
  File "/usr/local/lib/python3.4/dist-packages/httpie/client.py", line 48, in get_response
    response = requests_session.request(**kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/adapters.py", line 477, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

For reference, curl works fine: curl https://example_using_my_ca.com

[ghost]

Would it make sense to use ssl.get_default_verify_paths() to get the default paths?

[ghost]

I would propose following behaviour:

If --verify is passed a parameter other than no or yes, pass the parameter through to requests.

If --verify is set to yes:

1. if REQUESTS_CA_BUNDLE is set, pass True to requests verify.
2. elif environment variable ssl.get_default_verify_paths().openssl_cafile_env is set, pass that to requests verify
3. elif environment variable ssl.get_default_verify_paths().openssl_capath_env is set pass that to requests verify
4. elif ssl.get_default_verify_paths().capath is not None, pass that to requests verify
5. else pass True to requests verify

but please note that ssl.get_default_verify_paths is available only since python3.4

[sigmavirus24]

@luv not only is that only available in 3.4+ and 2.7.10+ but it doesnt' work on every platform. This is why requests is working on its own issue to resolve this for people. Please stop constantly commenting on httpie. This is a concern that exists in requests and not httpie IMO.

[ghost]

@sigmavirus24 wtf man? No idea what you mean by constantly commenting, but whatever, I see you are not even a httpie contributor. I assume just a stupid troll :/

[ghost]

a quick glance at the curl source code to see what a working implementation looks like ....

Disregarding all those amiga and VMS ifdefs and support for many different ssl libraries, it's actually pretty stupid and it is not using openssl X509_* lookup methods (as, for example, used in get_default_verify_paths() in Python SSL module).

Instead, curl simply iterates over a bunch of known locations at the compile time (see this m4dness https://github.com/curl/curl/blob/master/acinclude.m4#L2560 ) and then explicitly supports overriding with (CURL_CA_BUNDLE and) SSL_CERT_DIR and SSL_CERT_FILE environment variables at runtime (again not using stuff like X509_get_default_cert_file_env()). That's it.

So what about implementing the same approach in pure python? (Yes it looks quick and dirty as hell but it works for curl!) but adding support for windows (ssl.enum_certificates ?) and OS X (not sure here but python seems to use the apple-supplied openssl library on OS X 10.6+ so that should be fine already!).

[jkbrzt]

@luv thanks for the report but please be more respectful to other members. @sigmavirus24 is a core developer of requests on which httpie relies for all HTTP and without which it wouldn't even exist, so his view is extremely relevant.

[ghost]

please note this wont get fixed in requests until 3.0.0 release ... not even SSL_CERT_FILE is going to be supported because of possible compatibility issues https://github.com/kennethreitz/requests/pull/2903

btw. I didn't mean to be disrespectful I really thought s/he was trolling :) I went out of my way to help find a fix for an issue and someone who has not even once contributed to the project (I checked HTTPie's contributors because I was really suprised by such a hostile reaction) was basically telling me to shut up :/

[sigmavirus24]

@luv I was not telling you to shut up. There are 763 people subscribed to this issue. 100% of them might receive emails for it (but it's probably more like 70% or ~534). That means you generated (in the 2 hours after the issue was opened) ~1602 emails (possibly more or less). In other words, you're spamming people with stream of conscious posts like this. The best way to document new information when no one has replied to you is by editing the original post.

Finally code contributions are not the only contribution to a project. If you want to see all the contributions on a project you might use a project like octohatrack. I've responded to several requests related bugs on httpie for @jkbrzt because they don't quite have the time to follow requests' development. When I see a requests related bug here, I respond because 99% of the time, it's already being dealt with. Luckily for @jkbrzt, I've been treated as poorly (and a few times worse) before so I have thicker skin.

As a side note, "they" is as many characters as "s/he" and far more generally applicable since it can be used to refer to a single person as well.

[ghost]

so are we getting this fixed?

[sigmavirus24]

@jkbrzt I would call this a feature, not a bug. It's intentional that Requests works identically across Windows, *nixes, and BSDs and in fact, many system distributions of Requests actively remove that behaviour and point towards the system certificate store/bundle. So if users need this behaviour, they can use the version of Requests packaged by their system distributors with HTTPie (and possibly the system packaged HTTPie).

As for "getting this fixed", it depends on your definition of fixed. This a feature that will be added to Requests. At which point HTTPie will get the behaviour for free. If HTTPie instead deems this a higher priority (which @jkbrzt may do as they see fit) they can duplicate development effort to create a release which does this sooner.

As it seems you're using linux, @luv, you can take advantage of the fact that your distribution of linux almost certainly has a version of Requests that uses the system certificate store today. Requests is packaged in every distribution of linux I have checked and pretty much every one of those distributions uses their system certificate store. If that is not the case, perhaps you should be filing a bug with the Requests package maintainer of that distribution.

[ghost]

same prob with HTTPie installed via apt (ubuntu 14.04lts)

[sigmavirus24]

@luv that's surprising because I know for a fact that requests on 14.04 and 16.04 use the system certificate store. How did you install Requests?

[ghost]

I purged all "requests" and "httpie" versions to verify the ubuntu distribution of httpie is used and you are right I was still using httpie from PyPI. Ubuntu httpie fails with "ImportError: cannot import name is_windows" which is a known-issue.

I think I forgot to run "pip3 uninstall" and ran only "pip uninstall" as I had to use python3 to get working ssl in the first place.

I apologize to 761 persons to "spam" them with description of my httpie setup.

[powerman]

I'm using Gentoo Linux, httpie-0.9.9 and requests-2.18.4 (latest available in Gentoo) installed system-wide by package manager, no other versions installed (I don't even have pip installed). Other tools (openssl s_client, sslclient, curl) detect installed local CA cert and works fine, but httpie fails:

$ openssl s_client -quiet -connect localhost:8082                                              
depth=1 CN = Local CA home.lan
verify return:1
depth=0 CN = localhost
verify return:1
^C
$ http -v https://localhost:8082/                                                              

http: error: SSLError: HTTPSConnectionPool(host='localhost', port=8082): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),)) while doing GET request to URL: https://localhost:8082/

Am I should update something to make it work?

[sigmavirus24]

@powerman That's a more appropriate question for the gentoo redistributors of those packages.

[powerman]

@sigmavirus24 Why is that? My question is these requests/httpie versions are supposed to handle system CA list or not. If yes, then probably something is misconfigured on my system or there is a bug. If no, then which version I need to make it work.

[sigmavirus24]

@powerman nothing has changed. Neither supports your system certificate bundles but if you've installed both from your distribution's package manager then it's likely they've patched the software to use them. If you used the packages available in Gentoo and they're not using system certificates, then it's a Gentoo problem.

[openshiftninja]

Just going to add one additional comment on this: I use a proxy that is specifically designed so that when you make SSL requests, it establishes a secure tunnel between you and the proxy and then another one between the proxy and the destination. It does this so that I can securely monitor traffic going through my proxy to ensure no one is trying to do something nasty and hide it. This unfortunately means that the certificate httpie gets back is not trusted (because it's in the system root, but httpie doesn't use it).

[suhr]

I can actually reproduce the same issue on NixOS.

[ghost]

4+ years later ...

[victordidenko]

Is there any workaround so far?
I started using mkcert tool, and was really upset to figure out, that httpie ignores system-wide installed CA :(
Specifying --verify="$(mkcert -CAROOT)/rootCA.pem" option is working fine, but I wouldn't really want to memorize that.

UPD:
Just made it brutally

cat "$(mkcert -CAROOT)/rootCA.pem" >> /usr/local/Cellar/httpie/2.4.0/libexec/lib/python3.9/site-packages/certifi/cacert.pem

not a nice way for certain, but does the job

[djeikyb]

Hack update for today's version of httpie, intel macos:

cat "$(mkcert --CAROOT)/rootCa.pem" >> /usr/local/Cellar/httpie/2.6.0/libexec/lib/python3.10/site-packages/certifi/cacert.pem

[jkbrzt]

@isidentical do you have an idea how to fix this directly in HTTPie?

[djeikyb]

@jkbrzt I would call this a feature, not a bug. It's intentional that Requests works identically across Windows, *nixes, and BSDs and in fact, many system distributions of Requests actively remove that behaviour and point towards the system certificate store/bundle. So if users need this behaviour, they can use the version of Requests packaged by their system distributors with HTTPie (and possibly the system packaged HTTPie).

As for "getting this fixed", it depends on your definition of fixed. This a feature that will be added to Requests. At which point HTTPie will get the behaviour for free. If HTTPie instead deems this a higher priority (which @jkbrzt may do as they see fit) they can duplicate development effort to create a release which does this sooner.

As it seems you're using linux, @luv, you can take advantage of the fact that your distribution of linux almost certainly has a version of Requests that uses the system certificate store today. Requests is packaged in every distribution of linux I have checked and pretty much every one of those distributions uses their system certificate store. If that is not the case, perhaps you should be filing a bug with the Requests package maintainer of that distribution.

I think psf/requests#2966 is the requests lib issue to follow along with?