docker-compose build -> SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

[PavelPolyakov]

!!! SOLUTION !!!
(thanks for @pmahoney docker/compose#890 (comment))

echo $CURL_CA_BUNDLE
unset CURL_CA_BUNDLE
docker-compose up

!!! END OF SOLUTION !!!

Hi,

As I was advised in docker/compose#890 (comment) (thanks @posita for the help), I want to raise a new issue that I can not make docker-compose works on my mac.
It is worth to state, that I have python and OpenSSL installed via brew, every package is up to date.

I have tried to install docker & co in two ways:

1. using brew
2. using *.pkg

But everytime I got the SSL issue.

I have the next setup currently:

OpenSSL 0.9.8zg 14 July 2015
docker-compose version: 1.4.0
docker-machine version 0.4.1 (e2c88d6)
Python 2.7.10

And that's what I have when doing docker-compose:

As @posita suggested, the issue has place because of the certificates misconfiguration:
docker/compose#890 (comment)
And the certificates are created during docker-machine provisioning and creation, so I have created an issues here.

However, I am not a specialist in this area :( But I want to have docker-compose work on my Mac still.
As I understand, the most full debug information is the result of the next command:

(
set -x
eval $( docker-machine env shiny-new-machine-74d5a19e )
env | grep DOCKER
ls -al "${DOCKER_CERT_PATH}"
openssl x509 -in "${DOCKER_CERT_PATH}/cert.pem" -text
openssl x509 -in "${DOCKER_CERT_PATH}/ca.pem" -text
docker-compose --verbose version
docker-compose --verbose ps
DOCKER_TLS_VERIFY=0 docker-compose --verbose ps
) >"${HOME}/Desktop/docker-compose-890-outerr-$( date -u +%Y-%m-%dT%H:%M:%SZ ).txt" 2>&1

Here you can find it: http://pastebin.com/vWqZgVKi

Any advice is appreciated.

Thanks for the help!

Regards,

[PavelPolyakov]

Any thoughts are appreciated, I'm stuck, and do not want to reinstall my OS to make docker-compose work :)

[ehazlett]

Thanks for reporting. Are you happening to be using a boot2docker vm with machine?

[posita]

This may be a red herring, but one thing I noticed is that docker-machine+boot2docker does not rely on boot2docker's default generated certs. Rather, machine seems to generate its own. When it does, the Subject of cert.pem is the same as the Issuer (i.e., the same as the Subject of ca.pem).

Due to a bug with certain (recent) versions of OpenSSL, this is substantially the same behavior that caused similar errors when using docker tools directly with boot2docker (i.e., without docker-machine; see also boot2docker/boot2docker#1029 and SvenDowideit/generate_cert#10). A fix was released with boot2docker 1.8.0 (@eaaaeae). However, as I mentioned, the (potentially) problematic behavior seems to be preserved when using machine, due to machine's own certificate handling.

My recommendation is to patch machine to make sure that the Subject of cert.pem differs from ca.pem and then have @PavelPolyakov test it in his environment. If the patched machine makes the error go away for him, I think you've got your fix. If not, then it's something else that is beyond my understanding of the problem.

I would submit a PR, but I am not (yet) very skilled with Go, nor am I familiar with how machine handles certificate creation/installation with boot2docker. Unfortunately my current schedule does not allow me to dedicate the requisite time to come up to speed. In lieu of that, I hope this summary is useful.

[PavelPolyakov]

@ehazlett
Sorry, I haven't understood the question.

Here is how the machine was created:

docker-machine create --driver virtualbox my-docker-machine

The latest image was downloaded from the internet.

@posita
Thanks for the help, ready to test anything which could lead to success.

[PavelPolyakov]

@ehazlett
Hi guys, could you tell me if I can help somehow? And how we can make it working?

Regards,

[PavelPolyakov]

@RKLT

Tried that:

Seems like the result is the same.

Any other thoughts?

[rkit]

@PavelPolyakov Sorry, I was wrong…
I am looking for a solution ...

[rkit]

@PavelPolyakov Done. That's what did:

1. Removed all machines
2. Run docker-machine regenerate-certs
3. Add

export DOCKER_HOST=tcp://192.168.99.100:2376
export DOCKER_CERT_PATH=/Users/$USERNAME/.docker/machine/certs
export DOCKER_TLS_VERIFY=1

The error was gone.

[PavelPolyakov]

@rkit
Thanks for the solution.

Trying to implement it.

However, couple of questions:

1. When I have removed the machine, I think I am not able to regenerate-certs.

[~/tmp/microservices-workshop/msworkshop/step4]$ docker-machine rm default
Successfully removed default
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-machine regenerate-certs                   *[master]
Regenerate TLS machine certs?  Warning: this is irreversible. (y/n): y
Regenerating TLS certificates
Error: Expected to get one or more machine names as arguments.

1. Shouldn't this IP be dynamic, and is available only after the docker-machine is created?

export DOCKER_HOST=tcp://192.168.99.100:2376
export DOCKER_CERT_PATH=/Users/$USERNAME/.docker/machine/certs
export DOCKER_TLS_VERIFY=1

[rkit]

1. Yes, you need to create a machine
2. Dynamic, yes. To change env, you need to execute: eval "$(docker-machine env dev)", but default env from bash profile, and I added this line in the config:

export DOCKER_CERT_PATH=/Users/$USERNAME/.docker/machine/certs

After that there was no error.

[posita]

Am I interpreting correctly, that requiring regenerate-certs is a work-around that confirms my speculation about the cause of this issue? I.E., that docker-machine has a sister issue to boot2docker/boot2docker#808 (see also docker/docker-py#465, docker/compose#890) which was not resolved with boot2docker/boot2docker#1029 because docker-machine ignores the boot2docker default certificates and creates its own? Is that an accurate summary?

[PavelPolyakov]

@rkit
Sorry, I have the same thing still :(

What I did:

1. docker-machine rm default
2. docker-machine create --driver virtualbox default
3. docker-machine regenerate-certs default
4. eval "$(docker-machine env default)" export DOCKER_CERT_PATH=/Users/$USERNAME/.docker/machine/certs

Have I missed something? Any thoughts how I can overcome it?
Can I provide some additional debug?

[posita]

@PavelPolyakov, after doing the regenerate-certs and eval, can you do:

for i in ca cert ; do c="${DOCKER_CERT_PATH}/${i}.pem" ; ( set -x ; openssl x509 -in "${c}" -text | grep -E '^ +(Issuer|Subject): ' ) ; done

[PavelPolyakov]

@posita
Thanks, here we are:

[posita]

Okay, so regenerate-certs doesn't actually address the (potential) problem where ca.pem Subject == cert.pem Subject. 😞 In other words:

+-zsh:xxx> openssl x509 -in /.../.docker/machine/certs/ca.pem -text
+-zsh:xxx> grep ... -E '^ +Subject: '
        Subject: O=[thing]    # <<<-- THIS SHOULD *NOT*
+-zsh:xxx> openssl x509 -in /.../.docker/machine/certs/cert.pem -text
+-zsh:xxx> grep ... -E '^ +Subject: '
        Subject: O=[thing]    # <<<-- EQUAL THIS

If it does, and you have the "wrong" version of OpenSSL (i.e., any version that exhibits this bug), you will receive the [SSL: CERTIFICATE_VERIFY_FAILED] error.

I suspect this is why you're still having issues, and why @rkit's suggestion does not work.

[PavelPolyakov]

Hi guys, @rkit , @ehazlett , any thoughts how I can overcome that?

[posita]

@PavelPolyakov, for giggles, can you try this (replace [machine-name] with your actual machine name)?

% eval $( docker-machine env [machine-name] )
% docker-machine ssh [machine-name]
...
Boot2Docker version 1.8.2, build master : aba6192 - Thu Sep 10 20:58:17 UTC 2015
Docker version 1.8.2, build 0a8c2e3
docker@boot2docker:~$ rm -fv ~docker/.docker/* # get rid of copies of certificates we're about to destroy
removed '/home/docker/.docker/ca.pem'
removed '/home/docker/.docker/cert.pem'
removed '/home/docker/.docker/key.pem'
docker@boot2docker:~$ sudo -s 
root@boot2docker:/home/docker# cat /var/lib/boot2docker/profile # so I can see what is being overridden
...
root@boot2docker:/home/docker# rm -fv /var/lib/boot2docker/*.pem /var/lib/boot2docker/tls/* # remove any existing certificates
...
root@boot2docker:/home/docker# /usr/local/etc/init.d/docker restart
... # should regenerate default certificates
root@boot2docker:/home/docker# exit
docker@boot2docker:~$ exit
% rm -v "${DOCKER_CERT_PATH}"/*.pem # remove host-side (stale) certificates
...
% docker-machine scp [machine-name]:/home/docker/.docker/\*.pem "${DOCKER_CERT_PATH}" # copy newly-created certificates from machine to host
...
% docker ps
...
% docker-compose ps
...
% openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -key "${DOCKER_CERT_PATH}/key.pem" -cert "${DOCKER_CERT_PATH}/cert.pem" -CAfile "${DOCKER_CERT_PATH}/ca.pem" -tls1 </dev/null
...

Basically, we're trying to avoid having docker-machine create the certificates, and instead rely on boot2docker's default mechanism. By logging in to the boot2docker instance, removing the certificates, and then restarting docker, we trigger that process. Then we replace the host-side certificates with the ones we just got boot2docker to (re)create.

[PavelPolyakov]

@posita

Can't regenerate certificates

First:

Boot2Docker version 1.8.2, build master : aba6192 - Thu Sep 10 20:58:17 UTC 2015
Docker version 1.8.2, build 0a8c2e3
docker@default:~$ rm -fv ~docker/.docker/*
removed '/home/docker/.docker/ca.pem'
removed '/home/docker/.docker/cert.pem'
removed '/home/docker/.docker/key.pem'
docker@default:~$ sudo -s
root@default:/home/docker# cat /var/lib/boot2docker/profile

EXTRA_ARGS='
--label provider=virtualbox

'
CACERT=/var/lib/boot2docker/ca.pem
DOCKER_HOST='-H tcp://0.0.0.0:2376'
DOCKER_STORAGE=aufs
DOCKER_TLS=auto
SERVERKEY=/var/lib/boot2docker/server-key.pem
SERVERCERT=/var/lib/boot2docker/server.pem


root@default:/home/docker# rm -fv /var/lib/boot2docker/tls/*
removed '/var/lib/boot2docker/tls/ca.pem'
removed '/var/lib/boot2docker/tls/cakey.pem'
removed '/var/lib/boot2docker/tls/cert.pem'
removed '/var/lib/boot2docker/tls/hostnames'
removed '/var/lib/boot2docker/tls/key.pem'
removed '/var/lib/boot2docker/tls/server.pem'
removed '/var/lib/boot2docker/tls/serverkey.pem'

Then:

root@default:~# /usr/local/etc/init.d/docker restart
Need TLS certs for default,127.0.0.1,10.0.2.15,192.168.99.104
-------------------
Generating CA cert
2015/09/25 18:06:48 Preventing overwrite: the following files already exist: "/var/lib/boot2docker/ca.pem". To overwrite files, add `--overwrite`.
rm: can't remove '/var/lib/boot2docker/server.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/server-key.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/cert.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/key.pem': No such file or directory
Generate server cert
/usr/local/bin/generate_cert --host=default,127.0.0.1,10.0.2.15,192.168.99.104 --ca=/var/lib/boot2docker/ca.pem --ca-key=/var/lib/boot2docker/tls/cakey.pem --cert=/var/lib/boot2docker/server.pem --key=/var/lib/boot2docker/server-key.pem --org=Boot2Docker
2015/09/25 18:06:48 Generating a server cert
2015/09/25 18:06:48 Failure to generate cert: open /var/lib/boot2docker/tls/cakey.pem: no such file or directory
Generating client cert
2015/09/25 18:06:48 no --host parameters, making a client cert
2015/09/25 18:06:48 Failure to generate cert: open /var/lib/boot2docker/tls/cakey.pem: no such file or directory
cp: can't stat '/var/lib/boot2docker/tls/cert.pem': No such file or directory
cp: can't stat '/var/lib/boot2docker/tls/key.pem': No such file or directory

[posita]

Ah, okay. I was afraid of that (/var/lib/boot2docker/profile overrides some default locations). Try the same thing, but instead of:

root@default:/home/docker# rm -fv /var/lib/boot2docker/tls/*

Do:

root@default:/home/docker# rm -fv /var/lib/boot2docker/*.pem /var/lib/boot2docker/tls/*

[PavelPolyakov]

@posita

rm -fv /var/lib/boot2docker/*.pem /var/lib/boot2docker/tls/*
removed '/var/lib/boot2docker/ca.pem'
removed '/var/lib/boot2docker/tls/hostnames'
root@default:~# /usr/local/etc/init.d/docker restart
Need TLS certs for default,127.0.0.1,10.0.2.15,192.168.99.104
-------------------
Generating CA cert
2015/09/25 18:18:14 Generating a new certificate authority.
rm: can't remove '/var/lib/boot2docker/server.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/server-key.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/cert.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls/key.pem': No such file or directory
rm: can't remove '/var/lib/boot2docker/tls//hostnames': No such file or directory
Generate server cert
/usr/local/bin/generate_cert --host=default,127.0.0.1,10.0.2.15,192.168.99.104 --ca=/var/lib/boot2docker/ca.pem --ca-key=/var/lib/boot2docker/tls/cakey.pem --cert=/var/lib/boot2docker/server.pem --key=/var/lib/boot2docker/server-key.pem --org=Boot2Docker
2015/09/25 18:18:15 Generating a server cert
Generating client cert
2015/09/25 18:18:15 no --host parameters, making a client cert
root@default:~# ls /var/lib/boot2docker/tls/
cakey.pem  cert.pem   hostnames  key.pem

(exit; exit)

but then:

[~/tmp/microservices-workshop/msworkshop/step4]$ docker ps           *[master]
An error occurred trying to connect: Get https://192.168.99.104:2376/v1.20/containers/json: x509: certificate signed by unknown authority
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose ps   *[master]
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

the last command:

openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -key "${DOCKER_CERT_PATH}/key.pem"
CONNECTED(00000003)
depth=0 O = default
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = default
verify error:num=21:unable to verify the first certificate
verify return:1
140735289209680:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1472:SSL alert number 42
140735289209680:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/O=default
   i:/O=PavelPolyakov
-----BEGIN CERTIFICATE-----
MIIDADCCAeqgAwIBAgIQR4h88WLsll8AkYK6MIeEMTALBgkqhkiG9w0BAQswGDEW
MBQGA1UEChMNUGF2ZWxQb2x5YWtvdjAeFw0xNTA5MjMxNjMwMDBaFw0xODA5MDcx
NjMwMDBaMBIxEDAOBgNVBAoTB2RlZmF1bHQwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQC7MYqKigy4aJCPiaCQrOKypICUYbIvLYtHKCYG7d9d274MuYWf
dH7mMKTTu8M3GOrAzUj+tvHDAHIZgV0nDMz5qwEkYI2JD7N7hcHQ91JUPJVVpV3e
v2v4hzFra8KB/9h4grjTwAnMLLJNNcWUwu8JPvy3xRmBjpl8UIVY1+VLYwiLodfS
7bWBdqYy2E0sxoJjgtNVPWgxkOloApOrQhq0t/HcyXv0mA3hVPG+9J1r5Mxqs0nY
/qIMKGpywhTGqsWfDnBe22Bf2tFHagDN0frLRrIrF8HT8Xggwawz9pN4RpRWUrI0
lAsgxznXZR1xnvffbxmdy2P94BX1UHqzVy/dAgMBAAGjUDBOMA4GA1UdDwEB/wQE
AwIAqDAdBgNVHSUEFjAUBggrBgEFBQcDAgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIw
ADAPBgNVHREECDAGhwTAqGNoMAsGCSqGSIb3DQEBCwOCAQEAYoGQ5x3wXGVxZey9
dcxyVnBkYSZiInVIeI+bdsKtxKUv7HoXRW1iLyKSeNq8BGQzpfaEPkDtfblcEctz
U0Oj7jS4MlrdzGK1A75Bl8Bo1VDpu146M8cp/Wa09tjmrJktMrVOVZVQteDCPmZo
RIgaHWH0gRJPOy2SCTvLhmyZTK0bO2YKpgvjf2f7NMw6GBJi1hgcZuw3wDjVutjS
2hOWh2HeVyguu1na+YTpZ0csH+MSrKt79ulBpZ3SlYiG5pDFJ5Bjf74OpL0KW30Z
sBA51T1y3Xfyohb27XT28mJLa5Q1cH1PTnComd4K9AlVy7zY0u/MlDN8kaZb0ggX
pqpv2Q==
-----END CERTIFICATE-----
---
Server certificate
subject=/O=default
issuer=/O=PavelPolyakov
---
Acceptable client certificate CA names
/O=PavelPolyakov
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1247 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID:
    Session-ID-ctx:
    Master-Key: C628B5966CBC7A394199AF9D0ECA66BF01744B865D7595D3F3080158A9C9AEDA72B53714D24FCFF11F542F38CBB5CC50
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1443205317
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

[posita]

Did you do this after exiting docker-machine ssh [machine-name] but before doing docker ps?

% rm -v "${DOCKER_CERT_PATH}"/*.pem
...
% docker-machine scp [machine-name]:/home/docker/.docker/\*.pem "${DOCKER_CERT_PATH}"
...

Keep in mind, you still need to replace your host-side certificates with the ones you just got boot2docker to (re)create before executing docker or docker-compose commands on the host. Otherwise, you'll get certificate mismatches like the ones you experienced.

By the way, you'll probably need to start over from the beginning of the steps I outline in my #1880 (comment). I've updated that comment to reflect the change in my #1880 (comment).

[PavelPolyakov]

In my $DOCKER_CERT_PATH I have new (I think so) certificates located:

[~/.docker/machine/certs]$ ls
ca.pem   cert.pem key.pem

However, docker ps says:

[~/.docker/machine/certs]$ docker ps
An error occurred trying to connect: Get https://192.168.99.104:2376/v1.20/containers/json: x509: certificate signed by unknown authority

[posita]

I'm assuming your screenshot was just to show the details of cert.pem, not that you have added any of these *.pem files to any of your OS X keychains, correct?

There is a mismatch somewhere. What does this give you right now?

openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile  "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem"

If you aren't getting output that looks something like the following, you probably need to start over from scratch with the steps in my #1880 (comment).

...
---
Certificate chain
 0 s:/O=Boot2Docker
   i:/O=Boot2DockerCA
...
---
Server certificate
subject=/O=Boot2Docker
issuer=/O=Boot2DockerCA
---
Acceptable client certificate CA names
/O=Boot2DockerCA
...

[PavelPolyakov]

Yes, this is just info, I haven't add them to the keychain. But they are located in my DOCKER_CERT_PATH.

However, I still have:

Certificate chain
 0 s:/O=default
   i:/O=PavelPolyakov

and etc. , will try to startover again, thanks for the help!

[cischmidt]

I am having the same problem. I've followed the steps listed above and have the following signed certs:

+ openssl x509 -in /Users/cschmid/.docker/machine/certs/ca.pem -text
+ grep -E '^ +(Issuer|Subject): '
        Issuer: O=Boot2DockerCA
        Subject: O=Boot2DockerCA
+ openssl x509 -in /Users/cschmid/.docker/machine/certs/cert.pem -text
+ grep -E '^ +(Issuer|Subject): '
        Issuer: O=Boot2DockerCA
        Subject: O=Boot2Docker

But when I try to run docker-compose I continue to get the error

Chriss-MacBook-Pro:docker cschmid$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
Chriss-MacBook-Pro:docker cschmid$ docker-compose ps
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

I'm running docker-compose 1.4.2, docker 1.8.2 and have openssl v1.0.1j_1 installed.

[posita]

@cischmidt, what does this give you (in the same env)?

openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile  "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null

[cischmidt]

An error response, but it may be because I have to connect to docker via localhost because of my Cisco VPN client:

Chriss-MacBook-Pro:docker cschmid$ openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile  "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null
CONNECTED(00000003)
50891:error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version:/SourceCache/OpenSSL098/OpenSSL098-52.40.1/src/ssl/s23_clnt.c:593:

My docker environment variables are:

DOCKER_HOST=tcp://127.0.0.1:2376
DOCKER_MACHINE_NAME=default
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/Users/cschmid/.docker/machine/certs

[posita]

@cischmidt, you're on OS X, correct? I don't understand your DOCKER_HOST setting. 😕

[cischmidt]

I am on Mac OS X. I port-forward from my virtualbox VM to localhost:2376 to get around routing problems that the VPN introduces every time I connect to it. Docker related commands work well using this configuration, and I remember using docker-compose a few months ago without problem (today is the first day for some time that I began using docker-compose again).

[posita]

@cischmidt, just to confirm, localhost:2376 points to your docker-machine VM running on your local machine? Is your docker-machine VM boot2docker (e.g., running in VirtualBox)?

@PavelPolyakov and @cischmidt, can you do the following after having run the steps above in my #1880 (comment)? There's no need to run through those steps again if you're still in the same environment and the machine is still up.

% docker-machine ssh [machine-name]
docker@boot2docker:~$ export DOCKER_CERT_PATH="${HOME}/.docker" DOCKER_TLS_VERIFY=1 DOCKER_HOST=tcp://127.0.0.1:2376
docker@boot2docker:~$ docker ps
...
docker@boot2docker:~$ openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -CAfile  "${DOCKER_CERT_PATH}/ca.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null
...
docker@boot2docker:~$ exit

[posita]

The stack traces look weird to me (they don't look like SSL errors). What do you get for docker-compose --verbose build?

[PavelPolyakov]

[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose  build                             *[master]
Building frontend...
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose --verbose build                    *[master]
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

[posita]

What do you get with docker-compose --verbose ps and python compose-debug.py --verbose ps? This is what I get:

% docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
% docker-compose --verbose ps
Compose version 1.4.1
Docker base_url: https://www.xxx.yyy.zzz:2376
Docker version: KernelVersion=4.0.9-boot2docker, Os=linux, BuildTime=Thu Sep 10 19:10:10 UTC 2015, ApiVersion=1.20, Version=1.8.2, GitCommit=0a8c2e3, Arch=amd64, GoVersion=go1.4.2
docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=False']})
docker containers -> (list with 0 items)
docker containers <- (all=True)
docker containers -> (list with 0 items)
docker containers <- (all=False, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=True']})
docker containers -> (list with 0 items)
docker containers <- (all=True)
docker containers -> (list with 0 items)
Name   Command   State   Ports
------------------------------
% python compose-debug.py --verbose ps
DEBUG   : Trying /.../.docker/config.json
DEBUG   : File doesn't exist
DEBUG   : Trying /.../.dockercfg
DEBUG   : Attempting to parse as JSON
DEBUG   : ...
INFO    : Compose version 1.4.1
Compose version 1.4.1
INFO    : Docker base_url: https://www.xxx.yyy.zzz:2376
Docker base_url: https://www.xxx.yyy.zzz:2376
INFO    : Docker version: KernelVersion=4.0.9-boot2docker, Os=linux, BuildTime=Thu Sep 10 19:10:10 UTC 2015, ApiVersion=1.20, Version=1.8.2, GitCommit=0a8c2e3, Arch=amd64, GoVersion=go1.4.2
Docker version: KernelVersion=4.0.9-boot2docker, Os=linux, BuildTime=Thu Sep 10 19:10:10 UTC 2015, ApiVersion=1.20, Version=1.8.2, GitCommit=0a8c2e3, Arch=amd64, GoVersion=go1.4.2
INFO    : docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=False']})
docker containers <- (all=True, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=False']})
INFO    : docker containers -> (list with 0 items)
docker containers -> (list with 0 items)
INFO    : docker containers <- (all=True)
docker containers <- (all=True)
INFO    : docker containers -> (list with 0 items)
docker containers -> (list with 0 items)
INFO    : docker containers <- (all=False, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=True']})
docker containers <- (all=False, filters={u'label': [u'com.docker.compose.project=...', u'com.docker.compose.oneoff=True']})
INFO    : docker containers -> (list with 0 items)
docker containers -> (list with 0 items)
INFO    : docker containers <- (all=True)
docker containers <- (all=True)
INFO    : docker containers -> (list with 0 items)
docker containers -> (list with 0 items)
Name   Command   State   Ports
------------------------------

[PavelPolyakov]

@posita
That's what I have:

[~/tmp/microservices-workshop/msworkshop/step4]$ docker ps                                         *[master]
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[~/tmp/microservices-workshop/msworkshop/step4]$ docker --verbose ps                               *[master]
flag provided but not defined: --verbose
See 'docker --help'.
[~/tmp/microservices-workshop/msworkshop/step4]$ python compose-debug.py --verbose ps              *[master]
Traceback (most recent call last):
  File "compose-debug.py", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/compose/cli/main.py", line 39, in main
    command.sys_dispatch()
  File "/usr/local/lib/python2.7/site-packages/compose/cli/docopt_command.py", line 21, in sys_dispatch
    self.dispatch(sys.argv[1:], None)
  File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 27, in dispatch
    super(Command, self).dispatch(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/compose/cli/docopt_command.py", line 24, in dispatch
    self.perform_command(*self.parse(argv, global_options))
  File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 57, in perform_command
    verbose=options.get('--verbose'))
  File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 79, in get_project
    self.get_client(verbose=verbose))
  File "/usr/local/lib/python2.7/site-packages/compose/cli/command.py", line 64, in get_client
    version_info = six.iteritems(client.version())
  File "/usr/local/lib/python2.7/site-packages/docker/client.py", line 837, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python2.7/site-packages/docker/clientbase.py", line 86, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/usr/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
    if 'timed out' in str(err) or 'did not complete (read)' in str(err):  # Python 2.6
TypeError: __str__ returned non-string (type Error)
[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose --verbose ps                       *[master]
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
[~/tmp/microservices-workshop/msworkshop/step4]$ python -V                                         *[master]
Python 2.7.10

Do you think it's an issue of my python?
Which version do you have?

I also found this issue:
https://github.com/kennethreitz/requests/issues/2524

Regarding the https and ulr library which python uses, I don't know if docker-compose uses the same lib though.

[posita]

@PavelPolyakov, try docker-compose --verbose ps, not docker --verbose ps.

I doubt it's a Python version thing. I'm running 2.7.10 as well. It could be a docker-py thing? Try python -c 'import docker ; print(docker.version)'.

What version of OpenSSL do you have?

which openssl
openssl version
which python
python -c 'import ssl ; print(ssl.__file__) ; print(ssl.OPENSSL_VERSION)'

[PavelPolyakov]

Here we are:

[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose --verbose ps                       *[master]
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)
[~/tmp/microservices-workshop/msworkshop/step4]$ python -c 'import docker ; print(docker.version)' *[master]
1.3.1
[~/tmp/microservices-workshop/msworkshop/step4]$ which openssl                                     *[master]
/usr/local/bin/openssl
[~/tmp/microservices-workshop/msworkshop/step4]$ openssl version                                   *[master]
OpenSSL 1.0.2d 9 Jul 2015
[~/tmp/microservices-workshop/msworkshop/step4]$ which python                                      *[master]
/usr/local/bin/python
[~/tmp/microservices-workshop/msworkshop/step4]$ python -c 'import ssl ; print(ssl.__file__) ; print(ssl.OPENSSL_VERSION)'
/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.pyc
OpenSSL 1.0.2d 9 Jul 2015

[posita]

How is docker-py getting installed on your machine? How did you upgrade (if you upgraded) docker-compose to the latest version?

[PavelPolyakov]

Currently all the installation was done from here: https://www.docker.com/toolbox .
I haven't updated the docker-compose.

I have tried different installation as well - from brew (same results, SSL error), but the one we are experimenting with is from toolbox pkg.

[posita]

Okay, your versions aren't that different from mine, except for docker-py (I don't know about requests):

% docker version --format='{{.Client.Version}}'
1.8.2
% docker-machine --version
docker-machine version 0.4.1 (e2c88d6)
% docker-compose --version
docker-compose version: 1.4.1
% openssl version
OpenSSL 1.0.2d 9 Jul 2015
% python -c 'import ssl ; print(ssl.OPENSSL_VERSION)' # just to make sure it's picking up the same version
OpenSSL 1.0.2d 9 Jul 2015
% python -c 'import docker ; print(docker.version)'
1.4.0
% python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.6.2
1.10.3

Have you tried installing/running docker-compose from a virtual environment (that really should have been our first step 😞)?

% virtualenv .venv
...
% . ./.venv/bin/activate # don't forget the dot; you're "sourcing" the file
% ./.venv/bin/pip install docker-compose
...
% rehash # necessary on some shells like zsh
% which docker-compose
.../.venv/bin/docker-compose
% docker-compose --version
docker-compose version: 1.4.2
% python -c 'import docker ; print(docker.version)'
1.4.0
% python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.7.0
1.10.4
% docker-compose --verbose ps
...?
% python compose-debug.py --verbose ps
...?
% docker-compose build
...?

[PavelPolyakov]

The same:

[~/tmp/microservices-workshop/msworkshop/step4]$ virtualenv .venv                                  *[master]
New python executable in .venv/bin/python
Installing setuptools, pip, wheel...done.
[~/tmp/microservices-workshop/msworkshop/step4]$ . ./.venv/bin/activate                            *[master]
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ ./.venv/bin/pip install docker-compose     *[master]
Collecting docker-compose
  Downloading docker-compose-1.4.2.tar.gz (82kB)
    100% |████████████████████████████████| 86kB 2.2MB/s
Collecting docopt<0.7,>=0.6.1 (from docker-compose)
  Downloading docopt-0.6.2.tar.gz
Collecting PyYAML<4,>=3.10 (from docker-compose)
  Downloading PyYAML-3.11.tar.gz (248kB)
    100% |████████████████████████████████| 249kB 1.4MB/s
Collecting requests<2.7,>=2.6.1 (from docker-compose)
  Downloading requests-2.6.2-py2.py3-none-any.whl (470kB)
    100% |████████████████████████████████| 471kB 839kB/s
Collecting texttable<0.9,>=0.8.1 (from docker-compose)
  Downloading texttable-0.8.3.tar.gz
Collecting websocket-client<1.0,>=0.32.0 (from docker-compose)
  Downloading websocket_client-0.32.0.tar.gz (192kB)
    100% |████████████████████████████████| 192kB 1.0MB/s
Collecting docker-py<1.4,>=1.3.1 (from docker-compose)
  Downloading docker-py-1.3.1.tar.gz (49kB)
    100% |████████████████████████████████| 53kB 3.7MB/s
Collecting dockerpty<0.4,>=0.3.4 (from docker-compose)
  Downloading dockerpty-0.3.4.tar.gz
Collecting six<2,>=1.3.0 (from docker-compose)
  Downloading six-1.9.0-py2.py3-none-any.whl
Building wheels for collected packages: docker-compose, docopt, PyYAML, texttable, websocket-client, docker-py, dockerpty
  Running setup.py bdist_wheel for docker-compose
  Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/85/15/68/da301b79c711f2eb9a2065f44f16dcef736862e9c334bf87c4
  Running setup.py bdist_wheel for docopt
  Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/0d/5c/a7/cb986749520c1950217b5d8405def5c18541322dbc411a80d1
  Running setup.py bdist_wheel for PyYAML
  Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/fa/db/f6/dee55793d344f1706dc4a5a693298f0115241d1085cc212364
  Running setup.py bdist_wheel for texttable
  Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/d6/44/0b/20a20bc6ab19b4a7f4a43fa67010cf5008140d8abab61d58d5
  Running setup.py bdist_wheel for websocket-client
  Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/48/45/ba/e955834950f99f1ca7a5778808d7bef1d4962edb1a4b14600a
  Running setup.py bdist_wheel for docker-py
  Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/18/b4/cb/62ce0b04a3055cf5d599f3cfd01213d8488e9b367fb9cfb238
  Running setup.py bdist_wheel for dockerpty
  Stored in directory: /Users/PavelPolyakov/Library/Caches/pip/wheels/52/29/66/0c53de7d30b0e2a838ba252f6db929e9cc3d528892e7d759d5
Successfully built docker-compose docopt PyYAML texttable websocket-client docker-py dockerpty
Installing collected packages: docopt, PyYAML, requests, texttable, six, websocket-client, docker-py, dockerpty, docker-compose
Successfully installed PyYAML-3.11 docker-compose-1.4.2 docker-py-1.3.1 dockerpty-0.3.4 docopt-0.6.2 requests-2.6.2 six-1.9.0 texttable-0.8.3 websocket-client-0.32.0
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ rehash                                     *[master]
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ which docker-compose                       *[master]
/Users/PavelPolyakov/tmp/microservices-workshop/msworkshop/step4/.venv/bin/docker-compose
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose --version                   *[master]
docker-compose version: 1.4.2
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ python -c 'import docker ; print(docker.version)'
1.3.1
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.6.2
1.10.3
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose --verbose ps                *[master]
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ python compose-debug.py --verbose ps       *[master]
ERROR   : SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
(.venv)[~/tmp/microservices-workshop/msworkshop/step4]$ docker-compose build                       *[master]
Building frontend...
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

For some reason, you have all the versions higher then mine (if I understand correct):

1. docker
2. request
3. urlib3

why? and could it have such dramatic effect?

[posita]

For some reason, you have all the versions higher then mine (if I understand correct): ... why?

I'm not sure. Try deactivate-ing your virtualenv, removing it, and recreating it, but with ./.venv/bin/pip install --no-cache-dir docker-compose instead (note the addition of --no-cache-dir).

... and could it have such dramatic effect?

Apparently, yes. You're at least getting the same cryptic error for python compose-debug.py --verbose ps as docker-compose --verbose ps (instead of the stacktrace you were getting before). So chalk that up to requests, I guess.

FYI, this is what I get from my virtualenv (and a fresh ~/.docker/machine directory):

% deactivate
% rm -frv ./.venv
...
% for i in $( docker-machine ls --quiet ) ; do docker-machine stop "${i}" ; done
...
% docker-machine ls
NAME     ACTIVE   DRIVER       STATE     URL   SWARM
% mv -v ~/.docker/machine ~/.docker/machine.bak
/.../.docker/machine -> /.../.docker/machine.bak
% docker-machine create --driver virtualbox testes
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
To see how to connect Docker to this machine, run: docker-machine env testes
docker-machine create --driver virtualbox testes  8.12s user 6.22s system 13% cpu 1:47.48 total
% docker-machine ls
NAME     ACTIVE   DRIVER       STATE     URL                         SWARM
testes            virtualbox   Running   tcp://192.168.99.100:2376
% eval $( docker-machine env testes )
% docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
% virtualenv .venv
...
% . ./.venv/bin/activate
% ./.venv/bin/pip install --no-cache-dir docker-compose
...
% rehash
% cat Dockerfile
FROM debian:stable
% cat docker-compose.yml
deb1:
    build: .

deb2:
    image: debian:stable
    links:
        - deb1
% docker-compose ps
Name   Command   State   Ports
------------------------------
% docker-compose --verbose build
Compose version 1.4.2
Docker base_url: https://192.168.99.100:2376
Docker version: KernelVersion=4.0.9-boot2docker, Os=linux, BuildTime=Thu Sep 10 19:10:10 UTC 2015, ApiVersion=1.20, Version=1.8.2, GitCommit=0a8c2e3, Arch=amd64, GoVersion=go1.4.2
Building deb1...
docker build <- (pull=False, stream=True, nocache=False, tag=u'test_deb1', rm=True, path='/...', dockerfile=None)
docker build -> <generator object _stream_helper at 0x10f29b5f0>
Step 0 : FROM debian:stable
stable: Pulling from library/debian
401015d2a1e5: Pull complete
315baabd82d5: Pull complete
library/debian:stable: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:2ee35f51e54da93075fe46631a28d84ef4e23eb4ca51e7a8ef9f9ba625e7f6be
Status: Downloaded newer image for debian:stable
 ---> 315baabd82d5
Successfully built 315baabd82d5
docker close <- ()
docker close -> None
deb2 uses an image, skipping

[PavelPolyakov]

Nope :(

... steps above were done as well ...
[~/tmp/docker-compose-fixing]$ for i in $( docker-machine ls --quiet ) ; do docker-machine stop "${i}" ; done
[~/tmp/docker-compose-fixing]$ docker-machine ls
NAME     ACTIVE   DRIVER       STATE     URL   SWARM
testes            virtualbox   Stopped
[~/tmp/docker-compose-fixing]$ mv -v ~/.docker/machine ~/.docker/machine.bak
/Users/PavelPolyakov/.docker/machine -> /Users/PavelPolyakov/.docker/machine.bak/machine
[~/tmp/docker-compose-fixing]$ docker-machine create --driver virtualbox testes
Creating CA: /Users/PavelPolyakov/.docker/machine/certs/ca.pem
Creating client certificate: /Users/PavelPolyakov/.docker/machine/certs/cert.pem
Image cache does not exist, creating it at /Users/PavelPolyakov/.docker/machine/cache...
No default boot2docker iso found locally, downloading the latest release...
Downloading https://github.com/boot2docker/boot2docker/releases/download/v1.8.2/boot2docker.iso to /Users/PavelPolyakov/.docker/machine/cache/boot2docker.iso...
Creating VirtualBox VM...
Creating SSH key...
Error creating machine: exit status 1
You will want to check the provider to make sure the machine and associated resources were properly removed.
[~/tmp/docker-compose-fixing]$ docker-machine create --driver virtualbox testes
Creating CA: /Users/PavelPolyakov/.docker/machine/certs/ca.pem
Creating client certificate: /Users/PavelPolyakov/.docker/machine/certs/cert.pem
Image cache does not exist, creating it at /Users/PavelPolyakov/.docker/machine/cache...
No default boot2docker iso found locally, downloading the latest release...
Downloading https://github.com/boot2docker/boot2docker/releases/download/v1.8.2/boot2docker.iso to /Users/PavelPolyakov/.docker/machine/cache/boot2docker.iso...
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
To see how to connect Docker to this machine, run: docker-machine env testes
[~/tmp/docker-compose-fixing]$ docker-machine ls
NAME     ACTIVE   DRIVER       STATE     URL                         SWARM
testes            virtualbox   Running   tcp://192.168.99.109:2376
[~/tmp/docker-compose-fixing]$ eval $( docker-machine env testes )
[~/tmp/docker-compose-fixing]$ docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[~/tmp/docker-compose-fixing]$ virtualenv .venv
New python executable in .venv/bin/python
Installing setuptools, pip, wheel...done.
[~/tmp/docker-compose-fixing]$ . ./.venv/bin/activate
(.venv)[~/tmp/docker-compose-fixing]$ ./.venv/bin/pip install --no-cache-dir docker-compose
Collecting docker-compose
  Downloading docker-compose-1.4.2.tar.gz (82kB)
    100% |████████████████████████████████| 86kB 16.0MB/s
Collecting docopt<0.7,>=0.6.1 (from docker-compose)
  Downloading docopt-0.6.2.tar.gz
Collecting PyYAML<4,>=3.10 (from docker-compose)
  Downloading PyYAML-3.11.tar.gz (248kB)
    100% |████████████████████████████████| 249kB 2.7MB/s
Collecting requests<2.7,>=2.6.1 (from docker-compose)
  Downloading requests-2.6.2-py2.py3-none-any.whl (470kB)
    100% |████████████████████████████████| 471kB 15.5MB/s
Collecting texttable<0.9,>=0.8.1 (from docker-compose)
  Downloading texttable-0.8.3.tar.gz
Collecting websocket-client<1.0,>=0.32.0 (from docker-compose)
  Downloading websocket_client-0.32.0.tar.gz (192kB)
    100% |████████████████████████████████| 192kB 20.1MB/s
Collecting docker-py<1.4,>=1.3.1 (from docker-compose)
  Downloading docker-py-1.3.1.tar.gz (49kB)
    100% |████████████████████████████████| 53kB 10.5MB/s
Collecting dockerpty<0.4,>=0.3.4 (from docker-compose)
  Downloading dockerpty-0.3.4.tar.gz
Collecting six<2,>=1.3.0 (from docker-compose)
  Downloading six-1.9.0-py2.py3-none-any.whl
Installing collected packages: docopt, PyYAML, requests, texttable, six, websocket-client, docker-py, dockerpty, docker-compose
  Running setup.py install for docopt
  Running setup.py install for PyYAML
  Running setup.py install for texttable
  Running setup.py install for websocket-client
  Running setup.py install for docker-py
  Running setup.py install for dockerpty
  Running setup.py install for docker-compose
Successfully installed PyYAML-3.11 docker-compose-1.4.2 docker-py-1.3.1 dockerpty-0.3.4 docopt-0.6.2 requests-2.6.2 six-1.9.0 texttable-0.8.3 websocket-client-0.32.0
(.venv)[~/tmp/docker-compose-fixing]$ rehash
(.venv)[~/tmp/docker-compose-fixing]$ cat Dockerfile
FROM debian:stable
(.venv)[~/tmp/docker-compose-fixing]$ cat docker-compose.yml
deb1:
    build: .

deb2:
    image: debian:stable
    links:
        - deb1
(.venv)[~/tmp/docker-compose-fixing]$ docker-compose ps
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

However, library versions are the same, basically:

(.venv)[~/tmp/docker-compose-fixing]$ python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.6.2
1.10.3
(.venv)[~/tmp/docker-compose-fixing]$ python -c 'import docker ; print(docker.version)'
1.3.1

[posita]

However, library versions are the same, basically:

(.venv)[~/tmp/docker-compose-fixing]$ python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.6.2
1.10.3
(.venv)[~/tmp/docker-compose-fixing]$ python -c 'import docker ; print(docker.version)'
1.3.1

Oops...you're right. That's my fault (I was getting my terminals confused). I was (mistakenly) partially copying version numbers from my terminal where I had done something like pip install --upgrade ...). 😩 In my working environment, I have the same as you:

% which python
.../.venv/bin/python
% python -c 'import docker ; print(docker.version)'
1.3.1
% python -c 'import requests.packages.urllib3 ; print(requests.__version__) ; print(requests.packages.urllib3.__version__)'
2.6.2
1.10.3

What do you get for this in your current environment?

( set -x ; openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -tls1 -CAfile "${DOCKER_CERT_PATH}/ca.pem" -cert "${DOCKER_CERT_PATH}/cert.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null ; echo "exit code: ${?}" ) | awk '$0 ~ /TLS session ticket:/ { ticket = 1; } !ticket || $1 !~ /^[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]$/ { print; }'

Also, can you try this (corrected)?

% deactivate
% virtualenv -p /usr/bin/python .venv2
...
% ./.venv2/bin/pip install --no-cache-dir docker-compose
% ./.venv2/bin/python -c 'import ssl ; print(ssl.__file__) ; print (ssl.OPENSSL_VERSION)'
...
% ./.venv2/bin/python -c 'import _ssl ; print(_ssl.__file__) ; print (_ssl.OPENSSL_VERSION)' # note the underscore-ssl
...
% ./.venv2/bin/docker-compose ps
...

[PavelPolyakov]

1st:

(.venv)[~/tmp/docker-compose-fixing]$ ( set -x ; openssl s_client -showcerts -connect "${DOCKER_HOST#tcp:\/\/}" -tls1 -CAfile "${DOCKER_CERT_PATH}/ca.pem" -cert "${DOCKER_CERT_PATH}/cert.pem" -key "${DOCKER_CERT_PATH}/key.pem" </dev/null ; echo "exit code: ${?}" ) | awk '$0 ~ /TLS session ticket:/ { ticket = 1; } !ticket || $1 !~ /^[0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f][0-9A-Fa-f]$/ { print; }'
+-zsh:111> openssl s_client -showcerts -connect 192.168.99.109:2376 -tls1 -CAfile /Users/PavelPolyakov/.docker/machine/machines/testes/ca.pem -cert /Users/PavelPolyakov/.docker/machine/machines/testes/cert.pem -key /Users/PavelPolyakov/.docker/machine/machines/testes/key.pem
depth=1 O = PavelPolyakov
verify return:1
depth=0 O = testes
verify return:1
DONE
CONNECTED(00000003)
---
Certificate chain
 0 s:/O=testes
   i:/O=PavelPolyakov
-----BEGIN CERTIFICATE-----
MIIC/zCCAemgAwIBAgIQHxlNz0mEt4afcgoccLhiGzALBgkqhkiG9w0BAQswGDEW
MBQGA1UEChMNUGF2ZWxQb2x5YWtvdjAeFw0xNTA5MjcxODE2MDBaFw0xODA5MTEx
ODE2MDBaMBExDzANBgNVBAoTBnRlc3RlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAOhSHinVHHH9BjwuGyNwMPlvyLQvJHngyUqbRx5L+NaOL6lTGinr
vXG+xosolY8TuTdTd9LZu25ccQMQ3AbwVV0XNklkqxUc1cQma9pZ/9VsmfOlHl+n
HkgFrJIhiV+GpsjbXy/oNiwkuNGIu0/DByPbf7NWVoTKMOUKyJBFQnqWE9VWLXA/
4BVnl/8VOFnCdGCqx8EkJEk0RC4h8No76M0Z/t32w2wj1Cj8BFzO5slGwwcFuOTt
TQNfOyicxg/KUPeeLEr0st9WenBOE90lzozecqeBLLEwUcpFFFdNrC79Gy4W9YEd
f60kNsAxFxlvKLg5svxNJPwBmiA2mam54GkCAwEAAaNQME4wDgYDVR0PAQH/BAQD
AgCoMB0GA1UdJQQWMBQGCCsGAQUFBwMCBggrBgEFBQcDATAMBgNVHRMBAf8EAjAA
MA8GA1UdEQQIMAaHBMCoY20wCwYJKoZIhvcNAQELA4IBAQCfOH1AxRjw+ZDS04S8
ne+uUwk7KW5Ws2lZELAzedB9Izz/QOUyN89GGSttPrlUP0YOmMbwXOqFyMVuqx7e
WEGcGH1dduCbvP7SELBPYrocvAEQPWLqyA5W7uRiJA9KipRe2r5bVer1bvADt6Ir
UuNpoqYWM5fG+X0LT3PbNCEaxC8Itzb4KnVVCHrE+Me931uvOahLWrcisM6eQEYS
asDytbK9KvKM86TVEkyWFEAzTc4VWCLeK3JTy6Tfi+86Yqvk0TTaoh1X6hjKnfWY
5xT7SOL6DsNBZHDAUG0W/2FDR8EYfOmKoDW4PH5F7Qyamd0fcgMBrwZ/uMqhXHAd
vIsF
-----END CERTIFICATE-----
---
Server certificate
subject=/O=testes
issuer=/O=PavelPolyakov
---
Acceptable client certificate CA names
/O=PavelPolyakov
Client Certificate Types: RSA sign, ECDSA sign
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2165 bytes and written 1378 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : ECDHE-RSA-AES256-SHA
    Session-ID: 9329F3198FC79ACBDF2CAB18200C02B19F9AFD84AF4462C7044878A6A16BF257
    Session-ID-ctx:
    Master-Key: 52FFF668D11FCE75BADCA40CAFB7F016B888E975D87E2010EAC9ADE5CC0EEB5B562074D822D7306991E149BC2DCA83E6
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:

    Start Time: 1443381037
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
+-zsh:111> echo 'exit code: 0'
exit code: 0

and 2nd:

(.venv)[~/tmp/docker-compose-fixing]$ deactivate
[~/tmp/docker-compose-fixing]$ virtualenv -p /usr/bin/python .venv2
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in .venv2/bin/python
Installing setuptools, pip, wheel...done.
[~/tmp/docker-compose-fixing]$ ./.venv2/bin/pip install --no-cache-dir docker-compose
Collecting docker-compose
  Downloading docker-compose-1.4.2.tar.gz (82kB)
    100% |████████████████████████████████| 86kB 1.9MB/s
Collecting docopt<0.7,>=0.6.1 (from docker-compose)
  Downloading docopt-0.6.2.tar.gz
Collecting PyYAML<4,>=3.10 (from docker-compose)
  Downloading PyYAML-3.11.tar.gz (248kB)
    100% |████████████████████████████████| 249kB 2.6MB/s
Collecting requests<2.7,>=2.6.1 (from docker-compose)
  Downloading requests-2.6.2-py2.py3-none-any.whl (470kB)
    100% |████████████████████████████████| 471kB 11.1MB/s
Collecting texttable<0.9,>=0.8.1 (from docker-compose)
  Downloading texttable-0.8.3.tar.gz
Collecting websocket-client<1.0,>=0.32.0 (from docker-compose)
  Downloading websocket_client-0.32.0.tar.gz (192kB)
    100% |████████████████████████████████| 192kB 12.5MB/s
Collecting docker-py<1.4,>=1.3.1 (from docker-compose)
  Downloading docker-py-1.3.1.tar.gz (49kB)
    100% |████████████████████████████████| 53kB 12.5MB/s
Collecting dockerpty<0.4,>=0.3.4 (from docker-compose)
  Downloading dockerpty-0.3.4.tar.gz
Collecting six<2,>=1.3.0 (from docker-compose)
  Downloading six-1.9.0-py2.py3-none-any.whl
Installing collected packages: docopt, PyYAML, requests, texttable, six, websocket-client, docker-py, dockerpty, docker-compose
  Running setup.py install for docopt
  Running setup.py install for PyYAML
  Running setup.py install for texttable
  Running setup.py install for websocket-client
  Running setup.py install for docker-py
  Running setup.py install for dockerpty
  Running setup.py install for docker-compose
Successfully installed PyYAML-3.11 docker-compose-1.4.2 docker-py-1.3.1 dockerpty-0.3.4 docopt-0.6.2 requests-2.6.2 six-1.9.0 texttable-0.8.3 websocket-client-0.32.0
[~/tmp/docker-compose-fixing]$ ./.venv/bin/python -c 'import ssl ; print(ssl.__file__) ; print (ssl.OPENSSL_VERSION)'
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.pyc
OpenSSL 0.9.8zg 14 July 2015
[~/tmp/docker-compose-fixing]$ ./.venv2/bin/python -c 'import _ssl ; print(_ssl.__file__) ; print (_ssl.OPENSSL_VERSION)'
/Users/PavelPolyakov/tmp/docker-compose-fixing/.venv2/lib/python2.7/lib-dynload/_ssl.so
OpenSSL 0.9.8zg 14 July 2015
[~/tmp/docker-compose-fixing]$ ./.venv2/bin/docker-compose ps
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)

[posita]

Okay, I've officially run out of ideas. I cannot for the life of me fathom why openssl s_client can work, and docker ps can work, but you're still getting SSL certificate verification errors with docker-compose across two different versions of Python and two different versions of OpenSSL. Can you do a straight Docker build (i.e., docker build .)?

[PavelPolyakov]

Yes, docker build works ok:

[~/tmp/docker-compose-fixing]$ docker build -t hhh .
Sending build context to Docker daemon 27.28 MB
Step 0 : FROM debian:stable
 ---> 315baabd82d5
Successfully built 315baabd82d5
[~/tmp/docker-compose-fixing]$ docker run hhh --verbose
Error response from daemon: Cannot start container 4843e6bbbbbdd363356cb5536f82ce8cfada0960ed865242ebfe825ba7eade50: [8] System error: exec: "--verbose": executable file not found in $PATH

But docker-compose build not :)

[~/tmp/docker-compose-fixing]$ docker-compose build deb1
Building deb1...
SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

Nevermind :) Thanks for the help and effort. Anyhow it is good to know that someone had docker-compose working on OSX, probably, as some point I would have it working as well!

[posita]

I suppose one silver lining is that I no longer think your issue is with docker-machine. 😥

This may be an issue with docker-compose, but it's not docker/compose#890. Since it seems so difficult to reproduce outside of your environment, I suspect that you'll need to do some serious digging on your end to properly diagnose this (e.g., using pdb, maybe even dtrace or wireshark to fully understand what's going on).

I wish I could be of more help. 😞

[nathanleclaire]

Closing this, please re-open as a new issue if you can provide a reproducible test case. Thanks

[herveDarritchon]

@PavelPolyakov

One very dirty solution that did the trick for me :

Declare in the /etc/hosts, your docker ip address as localhost like
192.168.99.100 localhost

and then you put in the DOCKER_HOST localhost instead of your IP address.
export DOCKER_HOST=tcp://localhost:2376

And finally, on your mac, you will have either docker and docker-compose working together.

But I admit, it's really dirty :d but if it could help for a time, it worth may be a try.

[posita]

For those who are new to this issue, but have arrived here from outside (e.g,. web search), see @pmahoney's docker/compose#890 (comment) regarding the CURL_CA_BUNDLE environment variable being a possible cause. (@PavelPolyakov, if you can verify this was the cause of your issue, please consider updating your OP so others don't have to read the entire commit history.)

[PavelPolyakov]

@posita
Done, thanks for the help, good to have this issue solved.

[hongyanh]

I ran into the same error when I was using docker-compose commands. I finally uninstalled docker-machine and use boot2docker to re-build a mv and problem solved.

My OS is El Capitan and it seems mv created by docker-machine had some certificate conflicts with my system.

[lcarva]

Because docker-compose is python based and uses the requests python package, you must also unset the REQUESTS_CA_BUNDLE environment variable to make this work:

unset CURL_CA_BUNDLE
unset REQUESTS_CA_BUNDLE
docker-compose up