Get "409 Conflict" but container not present

[alvinr]

Problem

Using compose to deploy containers into a swarm cluster on EC2 I got the following

docker-compose up -d
Creating prod_haproxy_1...
409 Conflict: Conflict. The name "prod_haproxy_1" is already in use by container 90d3cef6f3f3. You have to delete (or rename) that container to be able to reuse that name.

However, trying to remove the container failed with

docker rm -f 90d3cef6f3f3
Error response from daemon: Container 90d3cef6f3f3 not found
Error: failed to remove containers: [90d3cef6f3f3]

There does not appear to be a running container

docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                   NAMES
4183b9767e13        swarm:0.4.0-rc1     "/swarm join --advert"   3 hours ago         Host Down           2375/tcp                                swarm-4/swarm-agent
d2da7f71415d        swarm:0.4.0-rc1     "/swarm join --advert"   3 hours ago         Host Down           2375/tcp                                swarm-1/swarm-agent
ec9e4965a63e        swarm:0.4.0-rc1     "/swarm join --advert"   3 hours ago         Host Down           2375/tcp                                swarm-2/swarm-agent
2938b06b6d3d        swarm:0.4.0-rc1     "/swarm join --advert"   3 hours ago         Host Down           2375/tcp                                swarm-3/swarm-agent
fe329c8a2ed9        swarm:0.4.0-rc1     "/swarm join --advert"   3 hours ago         Host Down           2375/tcp                                swarm-0/swarm-agent
3fb10d7d55e6        swarm:0.4.0-rc1     "/swarm manage --tlsv"   3 hours ago         Host Down           2375/tcp, 52.1.100.214:3376->3376/tcp   swarm-0/swarm-agent-master

The logs of the daemon report the saem 409 error

ERRO[1676] Handler for POST /containers/create returned error: Conflict. The name "prod_haproxy_1" is already in use by container 90d3cef6f3f3. You have to delete (or rename) that container to be able to reuse that name. 
ERRO[1676] HTTP Error                                    err=Conflict. The name "prod_haproxy_1" is already in use by container 90d3cef6f3f3. You have to delete (or rename) that container to be able to reuse that name. statusCode=409

Workaround

Restarting the Engine seems to clear the problem, which seems to indicate that some state is getting reset.

Environment

uname -a
Linux swarm-0 3.16.0-43-generic #58~14.04.1-Ubuntu SMP Mon Jun 22 10:21:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker version
Client:
 Version:      1.8.0-dev
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   5fdc102
 Built:        Wed Jul 22 23:59:08 UTC 2015
 OS/Arch:      linux/amd64
 Experimental: true

Server:
 Version:      1.8.0-dev
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   5fdc102
 Built:        Wed Jul 22 23:59:08 UTC 2015
 OS/Arch:      linux/amd64
 Experimental: true

docker info
Containers: 3
Images: 80
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 86
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-43-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 7.304 GiB
Name: swarm-0
ID: L6LZ:YGL6:NLGC:XQRU:FNGW:WMCC:RILG:CWNC:DF26:NS5Q:X5YB:4WYZ
WARNING: No swap limit support
Labels:
 provider=amazonec2
 com.docker.network.driver.overlay.bind_interface=eth0
Experimental: true

[alvinr]

weirdly, the docker ps show the hosts as "down", but there is nothing in the logs apart from the "discovery" message every 20 seconds.

[cpuguy83]

I've found that swarm won't allow two containers with the same name even when on different nodes.

[cpuguy83]

Also, this is likely a swarm issue.

[alvinr]

@cpuguy83 thanks. Logged it in Engine since it appeared to come from there. I'll see if I can repo based on your info and open a Swarm issue (if there is not one already).

[cpuguy83]

@alvinr Any news?

[alvinr]

I need to re-run after the latest vendored in lib network - just to remove the possibility of a cascading issues. It does feel like there is some cached state being retained somewhere that is causing the root issue.

[anton-kasperovich]

Same for me, i tried to bake AMI with saved statement of docker-compose containers.

• docker-compose pull
• docker-compose up -d
• docker-compose stop

Then i baked Amazon AMI, used this AMI in my CloudFormation script to speed up creation of my services. But some containers can't start:

Recreating nexus
Service "nexus" is using volume "/sonatype-work" from the previous container. Host mapping "nexus_sonatype_work" has no effect. Remove the existing containers (with `docker-compose rm nexus`) to use the host volume mapping.
Device is Busy

or

WARNING: Service "nexus" is using volume "/sonatype-work" from the previous container. Host mapping "nexus_sonatype_work" has no effect. Remove the existing containers (with `docker-compose rm nexus`) to use the host volume mapping.
ERROR: Conflict. The name "/nexus" is already in use by container a72d2ddc9107c1066cf35c95e4fd45031be682fdd51df14ab1bbf26bd5953515. You have to remove (or rename) that container to be able to reuse that name.

But there is no container with ID - a72d2ddc9107c1066cf35c95e4fd45031be682fdd51df14ab1bbf26bd5953515

As @alvinr described, "service docker restart" helped me too.

Linux ip-10-0-1-51 3.14.35-28.38.amzn1.x86_64 #1 SMP Wed Mar 11 22:50:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker-compose version 1.6.0

Docker:

Client:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 19:55:25 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 19:55:25 2016
 OS/Arch:      linux/amd64

[robvelor]

Restarting the daemon seems to fix the issue. In my case I restarted the daemon on all the machines in the cluster. I will report back if I encounter this issue again.

[cpuguy83]

Closing as this has become multiple issues in one, and the original was most likely a swarm issue.

@robvelor For your issue, please see #23371