Can't stop docker container #35933
Comments
|
This really needs more information, and reproduction steps
|
|
No more messages are logged. Meanwhile, I made some more tests and after the container enters in this state:
To exit this state I have to:
I think I have reproduced this outside Jenkins one time but thought it was another problem. As with Jenkins, it is easier to reproduce this, I'll wait for next time to do docker inspect container. The setup (running in CENTOS vm):
And these steps are done for each test, and I'm running tests concurrently. I'm sorry for not giving more information, but this is what I could collect so far. |
|
I got a similar problem now with different docker version. I can't stop any container that is created. And this is logged for all containers. docker-info : docker version: docker inspect : |
|
@Timunas can you try updating to 17.12? |
|
The original issue was with 17.12 Regarding the original issue, I reproduced it once again and I cannot docker inspect it just hangs for all commands |
|
I get the same issue, though without using docker-compose. I'm using docker swarm. Same thing though, I occasionally get containers that neither docker swarm nor I with the docker CLI can stop. This causes docker swarm to end up collecting more replicas than desired that it can't scale down. Sometimes these replicas can still service requests and receive traffic. The only way to remove the containers is to restart docker on the effected node. |
|
I have the same issue with docker swarm. I remove one of multuple docker stacks, but only some of the containers in the stack are removed, while some containers hang around. Commands to I have the issue on an Azure VM: I also have the same issue on Docker for Mac (Edge: 17.12): |
|
We are also experiencing non-responsive docker-deamon on some commands: currently I cannot
this happends on multiple engines, all running 17.12. seems related to #35408 |
|
I experience the same bug. It is not consistent though. I don't see a pattern yet but it does happen. I am running Docker for Mac Version 17.12.0-ce-mac46 (21698). I am not running Docker in Docker. Container is created by Yes I can see that container is still running but (You can see that minutes passed before I pressed Ctrl-C) In another Terminal I tried to start another docker-compose project, that's what I have seen in the output the first time: Another project started fine but with these errors about stale file names above. Subsequent stops and starts of the another project did not throw any errors and worked fine. These files are on a named volume. The volume is mounted as Restarting Docker daemon solves the issue... temporarily. I forgot to do UPDATE: wanted to note that the container with issues has healthcheck on it. Looks like this might be the culprit. |
|
I get the same issue. Can reproduce it everytime using different environments: journalctl shows |
|
Also seeing this on 18.01. Hang on container inspect. |
|
We have stopped using 17.12 completely and rolled back to 17.09 because of this problem on 17.12 (macOS and apparently Linux as well). This is a critical, persistent problem. And unfortunately I have not found way to recreate it except using docker a lot. |
|
I'm experiencing the same issue in multiple servers using 17.12. As @rfay said, it didn't happen on 17.09. Checking the changelog, a major difference between 17.12 and 17.09 is that, since 17.11, Docker is based on containerd. So, as the evidences seem to indicate this is an issue in the runtime, maybe it would be good to investigate down this path. |
|
Yup, same here. I stick with 17.09 and recommend everyone using |
|
If you can grab a stacktrace from the running daemon it would be very helpful. I suspect, though, that this is the recent bug that was found in runc that is a race in handling the container I/O... which has been around since forever, apparently. |
|
The relevant runc patch is here, which you can try if you don't want to wait for a patched docker release: opencontainers/runc#1698 |
Please provide commands. I don't understand how to "hit" a relative url, and what is it relative to. I use Docker for Mac. What should I hit? |
|
@achekulaev or a TCP socket |
|
The following file is the output of that command ran in an AWS Ubuntu 16.04 instance using Docker version 17.12.0-ce, build c97c6d6 |
|
@ay0o Thanks! |
|
I took the logs on a MacBook Pro running macOS High Sierra 10.13.3, running docker 18.01.0-ce-mac48, channel: edge ee2282129d. |
|
@AlterEgo7 Thanks! This looks like docker is blocked in a syscall to write to disk, and even read from disk at least in one place. Seems like something is very wrong with the disk that is allocated for that docker VM in docker4mac. |
|
A number of i/o bound syscalls blocked for ~1 minute, actually. |
|
18.03.1 is out with some mitigations for this. Please let us know if it's still a problem on that release. |
|
@cpuguy83 Docker for AWS was still on 18.03.0 last time I checked. And last version listed in release notes is 18.03.0. I am eager to check. Any idea when Docker for AWS stable will upgrade to 18.03.1 ? |
|
Caution upgrading your swarm cluster, bug: #36961, your cluster can became dead. |
|
@cpuguy83 18.03.1 is not there yet at the release page: https://docs.docker.com/release-notes/docker-ce/ or am I blind? |
That doc are out of date, you can see here: https://github.com/docker/docker-ce/releases/tag/v18.03.1-ce |
|
@marcomsousa thanks for noticing that; release-notes are now also added on the docs website; https://docs.docker.com/release-notes/docker-ce/#18031-ce-2018-04-26 |
|
@cpuguy83 Is there a list somewhere of all of the issues related to this problem? That way we can know for sure when this issue is resolved and its safe to upgrade. |
|
@timdau This is mitigated by containerd/containerd@d235ae9 |
|
This commit containerd/containerd@d235ae9 was released in containerd 1.0.3. So we need to test if this error fixed in the 18.03.1 version |
|
Seems like 18.03.1 has fixed the issue for me. I have been using it for a week locally, but did not experience the issue, that was easily reproducible within a day otherwise. |
Update Docker to stable version which contains fix for moby/moby#35933 Docker-ce 18.03.1 includes the commit which fixes this containerd issue, per moby/moby#35933 (comment) And feedback based on comment: moby/moby#35933 (comment)
|
The 18.03.1 version seems to fixed this issue. (or mitigated was said @cpuguy83) I tested in 4 clusters. |
|
Thank you all for confirming; I'll go ahead and close this issue. If you still run into this on Docker 18.03.1 or above; please open a new issue with details |
|
@mavogel I had the same problem with freezing docker containers. The solution for me was that if I move logging from /dev/stderr to internal file inside docker container then the problem is gone. Probably there is some disk issue when container logs to /dev/stderr and probably it is the case for most of problems. |
|
My (temporary) solution in both version 18.06.1-ce and 18.09 was similar to @casperWWW. In my case I lowered the log level of the applications executed inside the containers and they stopped hanging. |
|
so what it seems is that the container cannot release allocated I/O resources. |
I 'm same with you,except restart docker on the effected node，any other way to solve the problem？ |
See my comment I've sent earlier here - #35933 (comment) |
- Should prevent issues mentioned here (moby/moby#35933) from happening. - Will update the version to latest once a stable release is out.
My docker version v17.12.1. |
|
Docker 17.12 has reached EOL over a year ago; are you able to reproduce on a current version? |
- Should prevent issues mentioned here (moby/moby#35933) from happening. - Will update the version to latest once a stable release is out.
17.12.1 has an issue where it will randomly freeze (see moby/moby#35933). If this resolves the issue running the tests on CI, we'll bump the minimum version required by batect itself as well.
and others
Description
Can't stop container.
I'm starting and removing containers concurrently using docker-compose.
Sometimes it fails to remove the containers.
I checked that I can't docker stop the container. The command hangs and after change docker daemon to debug I just see this line when I run the command.
dockerd[101922]: time="2018-01-04T15:54:07.406980654Z" level=debug msg="Calling POST /v1.35/containers/4c2b5e7f466c/stop"Steps to reproduce the issue:
Describe the results you received:
Can't stop container.
Describe the results you expected:
Container should have been stopped. And then removed.
Additional information you deem important (e.g. issue happens only occasionally):
Issue happens only occasionally
Output of
docker version:Output of
docker info:The text was updated successfully, but these errors were encountered: