-
Notifications
You must be signed in to change notification settings - Fork 18.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to remove a stopped container: device or resource busy
#22260
Comments
Same here. Exact same OS, also running on AWS (different instance types) with aufs. After stopping the container retrying docker rm several times and/or waiting a few seconds usually leads to "container not found" eventually. Issues exists in our stack at least since Docker 1.10. |
suffered from this issue for quite long time. |
Receiving this as well with Docker 1.10. I would very occasionally get something similar with 1.8 and 1.9 but it would clear up on it's own after a short time. With 1.10 it seems to be permanent until I can restart the service or VM. I saw that it may be fixed in 1.11 and am anxiously awaiting the official update so I can find out. |
"Device or resource busy" is a generic error message. "Me too!" comments do not help. @danielfoss There are many fixes in 1.11.0 that would resolve some |
I'm also seeing this problem on some machines and by taking a look at the code I think the original error is being obscured in here: https://github.com/docker/docker/blob/master/daemon/graphdriver/aufs/aufs.go#L275-L278 My guess is that the |
I tried to set my docker daemon in debug mode and got the following logs when reproducing the error:
I could find the message Version info:
|
I had the same problem using docker-compose rm
What I did to fix the problem without restarting docker :
It give you the pid of the processes which run in devices subsystem (what is mounted and busy) located in the hierarchy in /docker/:containerid: I succeeded to kill them : After their death, the container was gone (successfully removed) VersionClient: Server: |
There seems to be 2 different problems here as I am unable to fix my issue using @simkim's solution.
In my case, the cgroup associated with my container seems to be correctly deleted. The filesystem is also unmounted. The only solution for me is still to restart the Docker daemon. |
today same problem than @genezys
|
This appears to have gotten worse in 1.12... I have (some) idea of what may have caused this, but not quite sure of the solution (short of a revert). |
Yes I upgraded to 1.12 yesterday from 1.11 and now I got this problem two times in 2 days, never had it before on this host |
@genezys and myself are on debian 8, 3.16.7-ckt25-2+deb8u3 |
When @genezys and I run "docker-compose stop && docker-compose rm -f --all && docker-compose up -d", since docker 1.12 :
I tried to run all cron task during the day in case something was done during the night but it don't trigger the bug. |
Same information with more details, we can provide more information as requested as it append every morning. Stop and remove
Inspecting mount
Same set of process for the 4 containers Inspecting process
|
has anyone a better solution then restarting the docker service (version 1.12)? |
A workaround was proposed in #25718 to set So far, this has solved the problem for me. |
@genezys : Note the side effect of this workaround that I've explained in #25718 (comment) |
I was getting something like this: Error response from daemon: Driver aufs failed to remove root filesystem 6b583188bfa1bf7ecf2137b31478c1301e3ee2d5c98c9970e5811a3dd103016c: rename /var/lib/docker/aufs/mnt/6b583188bfa1bf7ecf2137b31478c1301e3ee2d5c98c9970e5811a3dd103016c /var/lib/docker/aufs/mnt/6b583188bfa1bf7ecf2137b31478c1301e3ee2d5c98c9970e5811a3dd103016c-removing: device or resource busy I simply searched for "6b583188bfa1bf7ecf2137b31478c1301e3ee2d5c98c9970e5811a3dd103016c" and found it was located in multiple folders under docker/ Hope it helps! |
The thing is, I can't remove that file. And lsof doesn't show any user of that file. I suspect this kernel bug so I just did |
I encouter same problem and i google for while. It seems the cadvisor container lock the file. |
@oopschen yes, that's a known issue; the cAdvisor uses various bind-mounts, including |
@thaJeztah Is there any solution or alternative for cadvisor? Thanks. |
@oopschen some hints are given in docker/docs#412, but it depends on what you need cAdvisor for to be able to tell what alternatives there are. Discussing alternatives may be a good topic for forums.docker.com |
Just got this error for the first time on OS X Sierra using docker-compose:
I had never seen it before the latest update last night.
I tried
In the amount of time it's taken me to type this out, the offending container is now gone.
I don't know if it's fixed itself, or if there's still a problem lurking... EDIT: Just started and stopped the same set of containers using |
@jeff-kilbride |
Docker 1.13 is an old version and reached end of life in March last year. Current versions of Docker should have fixes for this (but make sure your kernel and distro are up to date as well) |
@thaJeztah Can you point me to the url that has the end of line information? Docker 1.13.1 support overlay2 by default, why would it be end of life? |
@bamb00 https://docs.docker.com/install/#time-based-release-schedule
|
@thaJeztah I thought this was a kernel issue and not a docker issue. I'm running CentOS 7.4 - 3.10.0-862.3.2.el7.x86_64. |
It's a combination; on CentOS and RHEL, certain kernel features have been backported, but won't be enabled by default; current versions of docker take advantage of those features (in addition to many other improvements that prevent the issue) |
Has anyone been able to verify that this does not occur in EL7.4 or up? |
@xmj Haven't seen this issue since upgrade to 7.4 in November 2017. At the time was using Docker 17.06. Use 18.03 to be extra sure. |
With docker 1.13.1 and CentOS 7.5 (), Do I have to explicitly set /proc/sys/fs/may_detach_mounts to 1? I'm unable to upgrade docker from 1.13.1 for the time being. |
i have docker >18 ... i have same problem |
@publicocean0 Did you try these command?
|
@publicocean0 Did you try these command? cat /proc/mounts |grep docker ...then clear your /etc/sysconfig/docker-storage file. |
Hi @cpuguy83 I'm also getting the error in docker 1.12.6 running kernel 3.10.0-693.21.1.el7.x86_64
Unfortunately, I cannot afford to upgrade out of 1.12.6 for the time being. Is there a workaround like upgrading the kernel and/or using 1.12.6-cs13? Thanks in advance. |
umount the docker partition/volume. remove the fedora version. install
straight from docker.
…On Mon, Jun 18, 2018, 13:44 bamb00 ***@***.***> wrote:
Hi @cpuguy83 <https://github.com/cpuguy83>
I'm also getting the error in docker 1.12.6 running kernel
3.10.0-693.21.1.el7.x86_64
failed: [172.22.64.126] (item=/var/lib/docker) => {"changed": false, "item": "/var/lib/docker", "msg": "rmtree failed: [Errno 16] Device or resource busy: '/var/lib/docker/devicemapper/mnt/31d464385880ecb0972b36040ce912d3018fc91ba2b4f1f4cbf730baad7fa99c'
Unfortunately, I'm unable to upgrade out of 1.12.6 for the time being. Is
there a workaround like upgrading the kernel and/or using 1.12.6cs13?
Thanks in advance.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#22260 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AMqMOet0MyJjEFLtlN9Yqb-IVcZ0-WGFks5t9_UZgaJpZM4IN4Nz>
.
|
This need to be resolve from a production server. |
umount the partition/volume that docker_storage_setup created and you
should be able to remove that docker folder.
…On Mon, Jun 18, 2018, 15:11 bamb00 ***@***.***> wrote:
This need to be resolve from a production server.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#22260 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AMqMOe_D0K0kofXHYE0osGvzpve-AsJrks5t-AmJgaJpZM4IN4Nz>
.
|
It's racey, but you can set a PostStart command on the systemd unit to make
sure to set the graphdriver root (e.g. /var/lib/docker/devicemapper) to
"rshared"... `mount --make-rshared <driver root>`
…On Mon, Jun 18, 2018 at 1:32 PM LiverWurst ***@***.***> wrote:
umount the partition/volume that docker_storage_setup created and you
should be able to remove that docker folder.
On Mon, Jun 18, 2018, 15:11 bamb00 ***@***.***> wrote:
> This need to be resolve from a production server.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#22260 (comment)>, or
mute
> the thread
> <
https://github.com/notifications/unsubscribe-auth/AMqMOe_D0K0kofXHYE0osGvzpve-AsJrks5t-AmJgaJpZM4IN4Nz
>
> .
>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#22260 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAwxZoEY9okYZDln_SyqlrL20BSi5fLkks5t-A5ugaJpZM4IN4Nz>
.
|
You are referring to setting the 'mount --make-rshared /var/lib/docker/devicemapper' command in /etc/systemd/system/docker.service.d/ I wasn't sure if I need to create any arbitrary .conf file in /etc/systemd/system/docker.service.d to run the Poststart mount command. |
Yes, to make changes to a systemd unit, it's always recommended to create a "drop-in" ("override") file, and never modify the original unit-file; modifying the original file will cause it to not be updated if an updated version becomes available (i.e., when updating the version of docker you're running) |
@LiverWurst Can you explain what you mean by "remove the fedora version"? I'm running CentOS 7.3. Thanks. |
I had installed from the fedora repository with your basic dnf install
docker. I then removed via dnf remove docker and followed the instructions
@ docker.com/community-edition.
…On Mon, Jun 25, 2018, 19:02 bamb00 ***@***.***> wrote:
@LiverWurst <https://github.com/LiverWurst> Can you explain what you mean
by "remove the fedora version"? I'm running CentOS 7.3.
Thanks.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#22260 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AMqMOY9PDFrzIyY0VFEU_HpLDUQSWsf5ks5uAXoDgaJpZM4IN4Nz>
.
|
Restart the VM.. should work! |
docker rm -f embedded-java
Error response from daemon: container 8b91b15cb7939c05fb16cd26d13ee67bd33ca04af3a574193cee95f21e27ad2b: driver "aufs" failed to remove root filesystem: could not remove diff path for id 4119de17501b169eb0e4901dae4bc68e388d92a92f371ee53db9b93ec6970b2d: lstat /var/snap/docker/common/var-lib-docker/aufs/diff/4119de17501b169eb0e4901dae4bc68e388d92a92f371ee53db9b93ec6970b2d-removing/home/yocto/build/tmp/work/x86_64-linux/coreutils-native/8.30-r0/build/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3/confdir3: file name too long Solution(reboot not help, only rm via sudo): sudo su
rm -rf /var/snap/docker/common/var-lib-docker/aufs/diff/4119de17501b169eb0e4901dae4bc68e388d92a92f371ee53db9b93ec6970b2d-removing
docker rm -fv embedded-java |
If you get such error:
The solution here(No need to execute
|
This is now one of the top Google search results. If docker hangs in my experience it's usually because loses track of
If it still chokes up... If you have autostarting containers breaking it, then disable them while docker is off:
Disclaimer: not sure where else |
Apologies if this is a duplicate issue, there seems to be several outstanding issues around a very similar error message but under different conditions. I initially added a comment on #21969 and was told to open a separate ticket, so here it is!
BUG REPORT INFORMATION
Output of
docker version
:Output of
docker info
:Additional environment details (AWS, VirtualBox, physical, etc.):
Running on Ubuntu 14.04.3 LTS HVM in AWS on an m3.medium instance with an EBS root volume.
Steps to reproduce the issue:
$ docker run --restart on-failure --log-driver syslog --log-opt syslog-address=udp://localhost:514 -d -p 80:80 -e SOME_APP_ENV_VAR myimage
docker stop container
Error response from daemon: Driver aufs failed to remove root filesystem 88189a16be60761a2c04a455206650048e784d750533ce2858bcabe2f528c92e: rename /var/lib/docker/aufs/diff/a48629f102d282572bb5df964eeec7951057b50f21df7abe162f8de386e76dc0 /var/lib/docker/aufs/diff/a48629f102d282572bb5df964eeec7951057b50f21df7abe162f8de386e76dc0-removing: device or resource busy
$ sudo service docker restart
$ docker ps -a
shows that the container no longer exists.The text was updated successfully, but these errors were encountered: