Error response from daemon: client is newer than server with Docker 1.9 RC3

[arun-gupta]

Here is the version information:

> docker-machine -v
docker-machine version 0.5.0-rc3 (a1e610b)
> docker -v
Docker version 1.9.0-rc3, build 2100b94

Created a Docker machine:

> docker-machine create -d=virtualbox lab2
Running pre-create checks...
Creating machine...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
To see how to connect Docker to this machine, run: docker-machine env lab2

Configured as:

eval $(docker-machine env lab2)

docker ps gives the following error:

> docker ps
Error response from daemon: client is newer than server (client API version: 1.21, server API version: 1.20)

This is a newly created machine using the latest version of Docker CLI and Docker Machine.

[nathanleclaire]

Yeah, the UX is not great but if you want to use a RC Docker binary you need to specify to use a release candidate ISO to use e.g. --virtualbox-boot2docker-url https://github.com/tianon/boot2docker-legacy/releases/download/v1.9.0-rc4/boot2docker.iso

[arun-gupta]

Why this special treatment for RC?

This makes it non-intuitive.

[nathanleclaire]

Well, your Docker client binary is also an RC. How do you feel it should work?

[arun-gupta]

RC should automatically pick up the boot2docker.iso from RC. --virtualbox-boot2docker-url should only be used to override the default value. And the default should be the same as Docker client binary.

[nathanleclaire]

I do think we could do better about allowing upgrade / create to use new RCs by default, but currently RCs for boot2docker we've been keeping at @tianon's fork, so we need to change our habit of doing that as well if we're going to support this. I do worry a bit about making things too magic as everyone has really different expectations in this regard.

[arun-gupta]

Matching the boot2docker.iso with the Docker client binary seems like a reasonable and intuitive approach. And there will be an option to override anyway.

No magic, just intuitive at least to me!

[arun-gupta]

Seeing the exact same error with Docker 1.9.0 and Machine 0.5.0:

~ > docker ps
Error response from daemon: client is newer than server (client API version: 1.21, server API version: 1.20)
~ > docker -v
Docker version 1.9.0, build 76d6bc9
~ > docker-machine -v
docker-machine version 0.5.0 (04cfa58)

Don't see any way to reopen the issue now.

[nathanleclaire]

Did you run docker-machine upgrade?

[arun-gupta]

This machine was just created. Do I need to run docker-machine upgrade for that as well?

[nathanleclaire]

@arun-gupta Most likely, to update your cache and/or in case you made the machine before the official b2d release happened.

[arun-gupta]

@nathanleclaire created the machine 30 mins ago again and still got the same error. docker-compose up -d on a docker-compose.yml worked successfully. But docker ps is giving the error again:

> docker ps
Error response from daemon: client is newer than server (client API version: 1.21, server API version: 1.20)

Upgrading the machine explicitly.

Is there a mapping of b2d.iso and Client/Server API version?

[nathanleclaire]

docker-machine upgrade name should work, my guess is it's an "ISO cache issue". Did you try that command?

The boot2docker.iso always maps to the corresponding Docker release API version. You can see the cache's version in the metadata by doing file $HOME/.docker/machine/cache/boot2docker.iso.

[arun-gupta]

docker-machine upgrade resolved the issue.

[nathanleclaire]

Thanks Arun. We're going to be working on some issues around upgrade flow this iteration so hopefully it will be a little more clear in the future.

[lisp-ceo]

👍 for docker-machine upgrade

[brandontamm]

+1 for docker-machine upgrade - assuming the only thing being upgraded is docker-related ;)

[danaucpe]

👍 for docker-machine upgrade <machine name>

[jgsqware]

After upgrade with Docker Toolbox, default machine was stopped but upgraded.
Others machine was not stopped, and not upgraded.

docker-machine upgrade <machine-name>work also for me

[amar-sanakal]

I'm on Windows 10 and saw a similar issue which strangely just disappeared on a second run. Here's what I had and the sequence of steps.

1. I was using the Docker Toolbox 1.8.3 and decided to use the latest release 1.9.0c as I was facing some weird issues.
2. I ran a docker-machine rm default just as a safety step
3. Downloaded and installed the toolbox version 1.9.0c
4. Git was the only thing I did not pick when prompted and installed everything else.
5. Kicked off the Docker Quickstart Terminal
6. Everything looked fine

Creating Machine default...
Running pre-create checks...
Creating machine...
Waiting for machine to be running, this may take a few minutes...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Provisioning created instance...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
To see how to connect Docker to this machine, run: C:\Program Files\Docker Toolbox\docker-machine.exe env default
Setting environment variables for machine default...



                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""\___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           \______ o           __/
             \    \         __/
              \____\_______/

docker is configured to use the default machine with IP 192.168.99.100
For help getting started, check out the docs at https://docs.docker.com

7.Checked to see that machine is created

$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM
default   *        virtualbox   Running   tcp://192.168.99.100:2376

8.All fine so far, but

$ docker ps
Error response from daemon: client is newer than server (client API version: 1.21, server API version: 1.20)

9.What do I have?

$ docker-machine -v
C:\Program Files\Docker Toolbox\docker-machine.exe version 0.5.0 (04cfa58)
$ docker -v
Docker version 1.9.0, build 76d6bc9

10.Run the machine upgrade as suggested above and I get:

$ docker-machine upgrade default
Stopping machine to do the upgrade...
C:\Program Files\Oracle\VirtualBox\VBoxManage.exe showvminfo default --machinereadable failed:
VBoxManage.exe: error: The object is not ready
VBoxManage.exe: error: Details: code E_ACCESSDENIED (0x80070005), component ConsoleWrap, interface IConsole, callee IUnknown
VBoxManage.exe: error: Context: "COMGETTER(SharedFolders)(ComSafeArrayAsOutParam(folders))" at line 2306 of file VBoxManageInfo.cpp
$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL   SWARM
default   -        virtualbox   Stopped
$ docker-machine upgrade default
Error: machine must be running to upgrade.
$ docker-machine start default
Started machines may have new IP addresses. You may need to re-run the `docker-machine env` command.
$ docker-machine ls
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM
default   *        virtualbox   Running   tcp://192.168.99.100:2376
$ docker-machine upgrade default
Stopping machine to do the upgrade...
Upgrading machine default...
Latest release for github.com/boot2docker/boot2docker is v1.9.0

Downloading https://github.com/boot2docker/boot2docker/releases/download/v1.9.0/boot2docker.iso to C:\Users\ssluser\.docker\machine\cache\boot2docker.iso...
Starting machine back up...

11.All fine after that. Just running the upgrade a second time seemed to work.

[ericm24]

@arun-gupta @nathanleclaire docker-machine upgrade [machine-name] fixed mine!!! Thanks Much. FWIW, this upgrade-sync between client & server is really a PITA. Any cycles spent on making this better will be much appreciated!

[superdump]

This is not restricted to RC3 nor to docker machine. If the docker client is 1.9.x and the server is running docker 1.8.x, the error message is observed. This is very impractical in terms of managing deployments if you do not or cannot have the same docker engine version installed on all servers and all clients. I am of the opinion that this is quite a serious breakage.

[superdump]

Also basically the same issue as #1839

[pcerioli-gpsw]

docker-machine upgrade <machine-name> worked also for me

[johnjelinek]

docker-machine upgrade <machine-name> did not work for me. I had to upgrade all my servers to a dev build of docker, then I downgraded again and started getting:

docker: Error response from daemon: client is newer than server (client API version: 1.22, server API version: 1.21).

After the manual downgrade, I then ran docker-machine upgrade <machine-name>, but the error remains.

[johnjelinek]

my fault ... I needed to delete the newer docker binary from my path.

[WaughWaugh]

docker-machine upgrade worked for me, as well.

[hheimbuerger]

Here's how I got it running after getting the same error for the 1.10.0-rc1:

docker-machine create -d=virtualbox -virtualbox-boot2docker-url https://github.com/tianon/boot2docker-legacy/releases/download/v1.10.0-rc1/boot2docker.iso docker

[nathanleclaire]

@ctroncoso I see your point, but if I run time docker info communicating with a server in amazonec2 / us-east-1 it takes a little over 300ms -- wouldn't a back-and-forth handshake for every request double this amount of time and introduce a non-trivial amount of overhead?

At any rate, there's nothing Machine can do about this issue, so I suggest an issue be opened (search for existing ones first) on https://github.com/docker/docker to share your thoughts with upstream if desired.

[ctroncoso]

@nathanleclaire sure it would, but would you trade 20 x 300 (or 600)ms for an upgrade that is guaranteed to work? Also, it would be just for the first call. Maybe a "session key" can be generated on that first call that establishes that the handshake has already been made. and used on the following without re-handshaking. I'm sure that there are security issues that this might bring up, but I'd rather have a not-so-fast fail-proof system, than one that might introduce an unsuspected load of work just because ubuntu/fedora/centos didn't update its repos on time. I see that this is an docker-engine issue but machine is taking the blame.

I'll check on the issues on docker-engine. I think we've got a nice feature going here. I like it when issue talks end up in a constructive idea. Thanks for your time @nathanleclaire !!!

[ghost]

The client already queries the server for version. There should be no
reason that the client would ever send an API parameter that doesn't exist,
because the client SHOULD be aware of the params available for that
version. Same goes for endpoints.

I would believe that this approach would require deprecation of older
version at some point, perhaps based on age or version delta.

You simply cannot have the client choking on older versions though, it's
not an option for production level deployments. I have 3 whole machines
here and I have having this issue, image what would happen on dispersed
deployments.

On Thu, Apr 21, 2016 at 2:47 PM, Nathan LeClaire notifications@github.com
wrote:

All we need to do here is make the client support older versions the API.
Why that wasn't a design requirement is odd.

What happens when a client from the future sends an API parameter that
doesn't exist to a daemon which doesn't expect it? Or makes a request to an
endpoint which doesn't exist in older versions? How is the daemon supposed
to know about all the possible things a future client might request of it?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#2147 (comment)

[nathanleclaire]

The client already queries the server for version

Can you point me to where in the Docker client code this occurs?

[ghost]

I don't really know Go, but I'm fairly sure that's what this does:

https://github.com/docker/docker/blob/eab65e438ecc406baf935c8df544982164cff72f/integration-cli/docker_api_test.go

Either way you see a pattern of querying API version all over the project.

On Thu, Apr 21, 2016 at 5:25 PM, Nathan LeClaire notifications@github.com
wrote:

The client already queries the server for version

Can you point me to where in the Docker client code this occurs?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#2147 (comment)

[jiaxuanzhou]

here is my method to resolve this issue:
yesterday, when i made the newest version of docker from https://github.com/docker/docker.git referring to https://docs.docker.com/v1.5/contributing/devenvironment/ and modify the old docker client binary with the new one.
there occurred "client is newer than server with Docker",and the docker daemon failed to restart:

• /bin/systemctl status docker.service
  ● docker.service - Docker Application Container Engine
  Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)

but there are daemon binaries generated in the directory :
bundles/latest/binary-daemon
should add the directory to the PATH, or copy dockerd and containerd to the /usr/bin/
copy dockerd /usr/bin/docker
copy docker-containerd /usr/bin/containerd
copy ../binary-client/docker /usr/bin

then i modify /etc/init.d/docker to add the directory of the dockerd to PATH with the highlighted lines

DOCKERD=/home/master/github.com/docker/bundles/1.12.0-dev/binary-daemon
export PATH=$PATH:$DOCKERD

BASE=dockerd

modify these in /etc/default/$BASE (/etc/default/docker)
#DOCKER=/usr/bin/$BASE
DOCKER=$DOCKERD/$BASE
This is the pid file managed by docker itself
DOCKER_PIDFILE=/var/run/$BASE.pid
This is the pid file created/managed by start-stop-daemon
DOCKER_SSD_PIDFILE=/var/run/$BASE-ssd.pid
DOCKER_LOGFILE=/var/log/$BASE.log
DOCKER_OPTS=
DOCKER_DESC="Docker"

then i restarted the service of dockerd ,it start succefully:
root@master:~# service docker status
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2016-05-04 04:32:01 EDT; 17h ago
Docs: https://docs.docker.com

root@master:~# docker version
Client:
Version: 1.12.0-dev
API version: 1.24
Go version: go1.5.4
Git commit: 1c0edf6-unsupported
Built: Wed May 4 05:05:37 2016
OS/Arch: linux/amd64

Server:
Version: 1.12.0-dev
API version: 1.24
Go version: go1.5.4
Git commit: 1c0edf6-unsupported
Built: Wed May 4 05:05:37 2016
OS/Arch: linux/amd64
root@master:~#

all run happy

just for reference

[ghost]

Hi, can someone please help me?
I am having the same issue. I understand docker-machine upgrade would work but I am not using docker on Windows / MAC. I am using it on Linux.

I am following this instructions to make a test up for playing with docker content trust
https://docs.docker.com/engine/security/trust/trust_sandbox/

In the dockerfile provided, it takes the 1.12.0 image and then creates the image , so when I run the container it does not connect to my base machine because my base machine (Linux) has 1.11.0 docker and this has 1.12.0 , so I then changed the docker file and passed the 1.11.0-dev path to it and generated the image again. This time when I ran the container with this new file, docker version is 1.11.0-dev only but API version is still 1.24 but my base Linux has API version of 1.23.

How do I get rid of this? How do I reduce my container API version to 1.23 or else increase my base API version to 1.,24 so that there will no errors?

P.S: I tried upgrading my base linux docker version but still API version is 1.23 only.

[nathanleclaire]

@mkonakan

On Ubuntu, sudo apt-get install -y docker-engine will update the natively installed version of Docker (Caveat: This will only work if you've installed Docker using the official channels)

docker-machine upgrade, as you noted, will update any boot2docker.iso you have to the latest version

[ghost]

@nathanleclaire Hi, upgraded my docker engine on my base ubuntu and it has latest 1.11.1 client version and API version 1.23 , but the container I built has 1.24 API version and hence there is the issue. So, I am looking for the way how can I downgrade my API version in container or else how can I upgrade my API version on base machine from 1.23 to 1.24?

[nathanleclaire]

@mkonakan Your best bet is probably to compile Docker from source, drop the binary into the relevant location, and restart the daemon. If the container you have built is using such a version of Docker it's likely that there's a line in the Dockerfile doing something similar.

[ghost]

Solved. I copied the binary file from my base machine to container instead of the default file in that dockerfile which is getting the latest API version. Thanks.

[nathanleclaire]

👍

[megastef]

Why is this closed? Is the latest docker client able to interface with older docker daemons?

[nathanleclaire]

@megastef Not that I'm aware of, but that's an issue of the upstream project (https://github.com/docker/docker), so I'd suggest looking for forwards compatibility issues there if you'd like to discuss.

[fairytaleordemon]

i have the same problem,i try to use docker-machine upgrade name ,so pity that is doesn't work.i dont know if the network although use proxy,but i solved this problem.
1.find the toolbox from
2.download and install it ,then it will upgrade ur version.

[pcgeek86]

docker-machine upgrade didn't work in my scenario. It seems that the CoreOS Docker Host is stuck at version 1.22. My client is running 1.24. How can I resolve this?

[eddieajau]

@pcgeek86 try export DOCKER_API_VERSION=1.23 (see https://forums.docker.com/t/docker-for-mac-stopped-running-docker-related-commands/16153/6)

[LEChaney]

For the people that find this having the same issue on Windows, use $env:DOCKER_API_VERSION=1.23 to set the environment variable.

[mjiderhamn]

I was having this issue as well, with the Windows beta. docker-machine upgrade did not help.
One alternative solution is to add --engine-install-url https://test.docker.com to docker-machine create, which will initialize the machine with the latest release candidate of Docker.

Details:

> docker -v
Docker version 1.12.0-rc4, build e4a0dbc, experimental
> docker-machine -v
docker-machine version 0.8.0-rc2, build 4ca1b85

> docker-machine create --driver amazonec2 aws01
> @FOR /f "tokens=*" %i IN ('docker-machine env aws01') DO @%i
> docker ps
Error response from daemon: client is newer than server (client API version: 1.24, server API version: 1.23)
(Here we could have used SET DOCKER_API_VERSION=1.23)

> docker-machine create --driver amazonec2 --engine-install-url https://test.docker.com aws02
> @FOR /f "tokens=*" %i IN ('docker-machine env aws02') DO @%i
> docker ps
(Works!)

[replaid]

Can this one be fixed (or perhaps worked around) by adding the DOCKER_API_VERSION to the output of docker-machine env?

[jcroldan]

I solved the problem thanks to @eddieajau.

My docker client (DOCKER_API_VERSION=1.24) is Ubuntu linux and the docker server (DOCKER_API_VERSION=1.23) is at Carina by Rackspace BETA.

Just add export DOCKER_API_VERSION=1.23 to your client to make it work.

Thanks.

[kid1412z]

export DOCKER_API_VERSION=1.23 solved my problem. thanks to @eddieajau

[marcusradell]

Thank you @kid1412z it worked for me too as a quick fix.

[twlz0ne]

Back to the older version

brew uninstall docker
brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/ba2a4f0358395010e84346d2224378c74d76c527/Formula/docker.rb
brew pin docker

[FlorianHeigl]

omg. negotiating protocol versions isn't a thing that still needs to be invented. i've worked with software from the 90s that could handle this. yuck, really.

[thaJeztah]

@FlorianHeigl docker client 1.13 and higher does API version negotiation with the daemon; the minimum daemon version that it will fall back to is docker 1.12. For older daemons, the DOCKER_API_VERSION override is needed

[norcino]

@eddieajau The workaround of the environment variable DOCKER_API_VERSION=1.23 doesn't work anymore.
I use docker for Windows and I'm connecting to a docker server running on a NAS that I cannot update.

Windows:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:09 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:45:38 2017
 OS/Arch:      linux/amd64
 Experimental: true

Nas:

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   781516c
 Built:        Thu Aug  3 16:04:05 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   781516c
 Built:        Thu Aug  3 16:04:05 2017
 OS/Arch:      linux/amd64

Does anyone have an idea?

[thaJeztah]

@manuelsalvatori that's an issue in the docker 17.09 cli; it's fixed in 17.10 (see moby/moby#35008)., not yet backported to 17.09.

Be aware that docker 1.11 is end of life though, and has known vulnerabilities, among which a CVE in RunC that allows container processes to break out of the container and get access to the host (resolved in docker 1.12.6 and up, which ship with a patched RunC version https://github.com/moby/moby/releases/tag/v1.12.6)