Specify password for SSL client side certificate

[botondus]

As far as I know currently it's not possible to specify the password for the client side certificate you're using for authentication.
This is a bit of a problem because you typically always want to password protect your .pem file which contains the private key. openssl won't even let you create one without a password.

[botondus]

Something like:

requests.get('https://kennethreitz.com', cert='server.pem', cert_pw='my_password')

[sigmavirus24]

Pretty sure you're supposed to use the cert param for that: cert=('server.pem', 'my_password')

[t-8ch]

@sigmavirus24
The tuple is for (certificate, key). Currently there is no support for encrypted keyfiles.
The stdlib only got support for those in version 3.3.

[Lukasa]

Heh, @t-8ch, you accidentally linked to a file on your local FS. ;) Correct link.

[sigmavirus24]

Quite right @t-8ch. This is why I should never answer issues from the bus. :/

[Lukasa]

So the current consensus is we don't support this. How much work is it likely to be to add support in non-3.3 versions of Python?

[sdog869]

How hard would it be to throw an error on this condition? I just ran into this silly problem and it took two hours to figure out, it would be nice if it would throw an error, it currently just sits there looping. Thanks for the awesome library!

[Lukasa]

Wait, it sits where looping? Where in execution do we fail? Can you print the traceback from where we loop?

[sdog869]

It seems to hang right here:

r = requests.get(url,
auth=headeroauth,
cert=self.cert_tuple,
headers=headers,
timeout=10,
verify=True)

I tried turning the timeout out up or down to no avail, but I imagine it knows well before the timeout it can't use the cert. Thanks!

[Lukasa]

Ah, sorry, I wasn't clear. I meant to let it hang and then kill it with Ctrl + C so that python throws a KeyboardInterrupt exception, then to see where we are in the traceback. I want to know where in Requests the execution halts.

[maxnoel]

What's happening (or at least what I've seen in many cases) is that OpenSSL, upon being given a password-protected certificate, will prompt the user for a password. It shows up in no logs (because the prompt is directly printed), and it doesn't time out because it's waiting for a user to press enter.

Needless to say, it's cubmersome, dangerous behavior when the code is running on a server (because it'll hang your worker with no option for recovery other than killing the process).

Is there a way to make requests raise an exception in that case instead of prompting for a password, or is that completely out of your control and in OpenSSL's hands?

[sigmavirus24]

@maxnoel I'm pretty sure this is in OpenSSL's hands but if you can answer @Lukasa's question (the last comment on this issue) it would be very helpful in giving a definite answer regarding if there was anything we can do to help.

[jjguy]

You can confirm OpenSSL is blocking on stdin for the passphrase from the interactive python prompt:

>>> r = requests.get("https://foo.example.com/api/user/bill", cert=("client.crt", "client.key"))
Enter PEM pass phrase:
>>>

If you're running from a backgrounded process, I assume OpenSSL will block waiting on that input.

[maxnoel]

That's correct. Is there anything requests can do to prevent that from happening? Raising an exception when no password is given would be far more useful than prompting for stuff on stdin (especially in a non-interactive program).

[Lukasa]

I'm afraid that I don't know of any way. @reaperhulk?

[reaperhulk]

There are ways to stop OpenSSL from doing this, but I'm not sure if they're exposed by pyOpenSSL. Where does requests call pyopenssl to load the client cert? I can dig a bit.

[Lukasa]

@reaperhulk It's done from in urllib3, here.

[Lukasa]

We also do something very similar for the stdlib, which will be a whole separate problem.

[Lukasa]

So we can do this with PyOpenSSL using a patch like this. In the stdlib version, we need to use load_cert_chain with a password.

[telam]

Has this problem been solved? I'm currently running into this while trying to connect to an Apache server.

[Lukasa]

It has not.

[mikelupo]

What about PKCS#12 formatted (and encrypted) containers which could contain a client cert/key? Would this fall under the same feature request?

[Lukasa]

@mikelupo Yup.

[Altynai]

@telam @mikelupo
I have the same problem and Googled a lot, finally, I solved it by using pycurl.
In my situation, I use openssl to convert my .pfx file to .pem file which contains both cert & key(encrypted with pass phrase), then invoke the following code.

import pycurl
import StringIO

b = StringIO.StringIO()
c = pycurl.Curl()
url = "https://example.com"
c.setopt(pycurl.URL, url)
c.setopt(pycurl.WRITEFUNCTION, b.write)
c.setopt(pycurl.CAINFO, "/path/cacert.pem")
c.setopt(pycurl.SSLKEY, "/path/key_file.pem")
c.setopt(pycurl.SSLCERT, "/path/cert_file.pem")
c.setopt(pycurl.SSLKEYPASSWD, "your pass phrase")
c.perform()
c.close()
response_body = b.getvalue()

BTW, for security, it's better to not do hardcode for pass phrase

[maxnoel]

Of course. That said, the problem isn't really that a pass phrase is required -- it's that OpenSSL makes your program hang while waiting for someone to type a passphrase in stdin, even in the case of a non-interactive, GUI or remote program.

When a passphrase is required and none is provided, an exception should be raised instead.

[FirefighterBlu3]

if you use a default passphrase of '' for the key, openssl won't hang.
it'll return a bad password text. you can immediately alter your py flow
to then notify the user without that apparant stall

[ldkingvivi]

any plan to add this feature

[Lukasa]

We want to add it, but we have no schedule to add it at this time.

[vinitkumar]

@botondus I think I found a simpler way to achieve this with request library. I am documenting this for other people who are facing the issue.

I assume that you have a .p12 certificate and a passphrase for the key.

Generate certificate and private key.

// Generate the certificate file.
openssl pkcs12 -in /path/to/p12cert -nokeys -out certificate.pem
// Generate private key with passpharse, First enter the password provided with the key and then an arbitrary PEM password //(say: 1234) 
openssl pkcs12 -in /path/to/p12cert -nocerts -out privkey.pem

Well, we are not done yet and we need to generate the key that doesn't require the PEM password every time it needs to talk to the server.

Generate key without passphrase.

// Running this command will prompt for the pem password(1234), on providing which we will obtain the plainkey.pem
openssl rsa -in privkey.pem -out plainkey.pem

Now, you will have certificate.pem and plainkey.pem, both of the files required to talk to the API using requests.

Here is an example request using these cert and keys.

import requests
url = 'https://exampleurl.com'
headers = {
            'header1': '1214141414',
            'header2': 'adad-1223-122'
          }
response = requests.get(url, headers=headers, cert=('~/certificate.pem', '~/plainkey.pem'), verify=True)
print response.json()

Hope this helps:

cc @kennethreitz @Lukasa @sigmavirus24

[kennethreitz]

I have heard through the grapevine that Amazon does exactly this, internally.

[mkane848]

@ideasean Getting invalid credentials still. I should be pointing the load_cert_chain at a .pem file generated by the pfx_to_pem function written for the Temp File method, correct? It has the private key and the cert in it.

Since the .pfx works with Postman but it won't authenticate here, could that mean that something's going wrong in the conversion process?

[seghcder]

I did not use the temp file method. I used the DESAdapter approach pretty much as written in AnoopPillai's post on Sep1 above starting with -

I did try with that code change (code pasted below) and ended up with the same error that i got with the tempfile method.

I can't speak to the conversion process, but perhaps a good test is to try using the converted pem file with Postman?

Also note that I used the approach above because my pem file was encrypted / password protected, and Python requests currently does not support that. If your pem ends up being not password protected, then you should be able to use native requests per link (but then you will have an unprotected cert on your file system).

[mkane848]

@ideasean I broke down the .pfx as per this method and got a .pem file with Bag Attributes and Certificate as well as a .pem file with Bag Attributes and an Encrypted Private Key.

Still getting invalid credentials, I guess I'll try putting the certs through on Postman and seeing if they work but I can't figure out why I'm apparently unable to unpack this .pfx properly

I also tried the openssl command openssl pkcs12 -in <my_pfx>.pfx -out certificate.cer -nodes, and it's still giving me a 401 error when I change to it like so: context.load_cert_chain('certificate.cer')

[mkane848]

I installed the above-mentioned .cer and Postman doesn't even ask to use it when I make the API call (unlike the popup when it asks to use the .pfx), not sure how else I can make it use that specific cert since there's no "Certificates" panel in the settings like the docs say there is.

[seghcder]

You may be using the browser version of Postman, which doesn't include the cert panel, ssl validation disable etc. Try the full client to change certificate settings. You may want to continue this discussion on a different thread then, as we are a bit off topic.

[aiguofer]

@mkane848 saw your original comment where you were getting a ValueError: String expected. You might want to check pyca/pyopenssl#701 and urllib3/urllib3#1275.

I use my private pem with a password using this:

from requests.adapters import HTTPAdapter

from urllib3.util.ssl_ import create_urllib3_context

class SSLAdapter(HTTPAdapter):
    def __init__(self, certfile, keyfile, password=None, *args, **kwargs):
        self._certfile = certfile
        self._keyfile = keyfile
        self._password = password
        return super(self.__class__, self).__init__(*args, **kwargs)

    def init_poolmanager(self, *args, **kwargs):
        self._add_ssl_context(kwargs)
        return super(self.__class__, self).init_poolmanager(*args, **kwargs)

    def proxy_manager_for(self, *args, **kwargs):
        self._add_ssl_context(kwargs)
        return super(self.__class__, self).proxy_manager_for(*args, **kwargs)

    def _add_ssl_context(self, kwargs):
        context = create_urllib3_context()
        context.load_cert_chain(certfile=self._certfile,
                                keyfile=self._keyfile,
                                password=str(self._password))
        kwargs['ssl_context'] = context

[vog]

For your information, I just implemented PKCS#12 support for requests as a separate library:

• https://pypi.python.org/pypi/requests-pkcs12

The code is a clean implementation: it uses neither monkey patching nor temporary files. Instead, a custom TransportAdapter is used, which provides a custom SSLContext.

Any feedback and improvements are welcome!

Of course, I wish requests would provide this functionality directly, but until we are there, this library will alleviate the pain.

[candlerb]

It would be very nice if we could simply do this:

    cert=("cert.pem", "key.pem", "somepassphrase")  # separate cert/key

    cert=("keycert.pem", None, "somepassphrase")    # combined cert/key

...even if it only worked on python 3.3+. This would only be a minor addition to the API surface.

AFAICS, this would mean a small change to urllib3 so that HTTPSConnection accepts an optional password argument; this is passed down through ssl_wrap_socket, ending up with:

    if certfile:
        if password is not None:
            context.load_cert_chain(certfile, keyfile, password)
        else:
            context.load_cert_chain(certfile, keyfile)

Then it would be backwards-compatible, raising an exception only if you try to use a private key passphrase on an older platform that doesn't support it.

Note that the contrib/pyopenssl.py adapter already supports this extra argument to load_cert_chain, and so does python 2.7.

---

Aside: I am using AWS KMS to manage "secret" data, so I would load the key password at runtime from KMS, not hard-code it into the application.

[kennethreitz]

I personally wouldn’t be against this change, as I think it would greatly improve our user interface for many users across the board.

@sigmavirus24 any thoughts?

[vog]

@candlerb @kennethreitz Would it be acceptable to include the PKCS#12 case into that API as well?

cert=('keycert.p12', None, 'somepassphrase')

The distinction could be either by file extension (*.p12 versus *.pem), or by looking at the first bytes of that file.

[candlerb]

I don't have a problem with allowing requests to take a pkcs#12, as long as it can be done safely - and in my opinion that precludes writing the extracted private key to a temporary file.

Googling for Python pkcs#12, I find:

• Someone's code which writes out the private key
• Some other code which I think depends on pyOpenSSL to read in the pkcs#12. It returns the certificate and key as data items.

So doing this, I think it would be necessary to hook things up in such a way that the key/cert themselves are passed to OpenSSL, not the filenames containing those things. That sounds like a much bigger change.

If that's too hard, then it just means that the user has to convert pkcs#12 to PEM off-line, which is pretty straightforward (and can be documented).

[vog]

@candlerb As I wrote in my previous comment (#1573 (comment)), I already created a clean implementation that integrates well with requests.

So the problems you are describing are already solved.

Right now my implementation adds new pkcs12_* keywords arguments, to stay out of the way as much as possible.

But I think it should be integrated into the cert keyword argument instead, and my question is:

• Would that be acceptable in general?
• Would my concrete proposal cert=('keycert.p12', None, 'somepassphrase') be acceptable?
• How should we distinguish between PKCS#12 and PEM? (By file name suffix, or by file contents?)

(Moreover, I'd prefer to see that into requests rather than my separate requests_pkcs12 library. But given the age of this issue, I have little hope that this will go upstream anytime soon. However, if there was a concrete statement about which kind of implementation exactly is wanted, maybe I could adjust my implementation accordingly and propose a pull request.)

[sigmavirus24]

So, a few things:

1. I don't think we should take the cert keyword and expand it like this. It's implicitly structured data and people are already confused by the tuples in the files keyword. I think continuing a known-bad pattern is foolish.

2. I think that if anything, the pkcs12 adapter should be modified and upstreamed into the requests-toolbelt. I think it would be better to modify it to create the ssl_context once instead of storing the pkcs12 password in memory on that object.

I think there's still other work that needs doing before we can handle this in the more general case no matter what and that includes determining the right API for this for Requests 3.0.

[vog]

@sigmavirus24 Thanks for the feedback.

1. Okay, so let's keep the separate pkcs12_* keywords.
2. Yes, that's definitely worth improving. I created an issue tracker entry for that: Create ssl_context only once m-click/requests_pkcs12#2

How would the PKCS#12 TransportAdapter class be included into requests? Would that class simply be added to requests, or is there another way to include it on a "deeper" level, so it can be used without any request()/get()/... wrappers and without having to explicitly load that adapter?

[rashley-iqt]

My organization has a need to use PKCS12 certificates and is willing to make the necessary enhancements to your library in order to do so. Decrypting the .p12 files to .pem files is considered too much of a risk and it adds an extra step to deal with. We'd like to add functionality to generate and provide an appropriate ssl_context for a given session. Is this still functionality your team would be willing to accept assuming it is implemented properly?

[vog]

Just a quick reminder: A clean implementation has already been provided by our company, but as a separate adapter: https://github.com/m-click/requests_pkcs12

Feel free to reformat it into a pull request for requests itself.

Along the way, you might want to fix a minor issue: The ssl_context should not be held in memory for a whole session, but as shortly as possible, just for a single given connection. See also:

• Create ssl_context only once m-click/requests_pkcs12#2

In case you fix it along the way, it would be nice if you could provide it as a small pull request to https://github.com/m-click/requests_pkcs12 in addition to requests itself.

That way, all people who are using the requests_pkcs12 library right now would automatically benefit from that improvement as well, without having to switch to the (then improved) new API for requests itself.

[KevinVerre]

Yeah, https://github.com/m-click/requests_pkcs12 worked for me and did exactly what I wanted it to do. Thanks so much @vog ! I hope requests is able to support that eventually.

[dzmitry-kankalovich]

I am also going to thank @vog for his implementation, works just as expected, and solves the problem of keeping cert/key in the non-secure storages like S3 in my case. Hopefully, this can make its way to requests.

[sethmlarson]

Closing as duplicate of #2519