No space left on machine (overlay)

[euqen]

I have docker running on CentOS Linux release 7.0.1406 (Core). Also I'm using overlay instead of devicemapper loopback and I have a few containers running on machine including docker registry container. After few days I had no space left on machine, therefore I do following:

[root@m13 lib]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        59G   53G  3.1G  95% /
devtmpfs        2.0G     0  2.0G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           2.0G  153M  1.8G   8% /run
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           395M     0  395M   0% /run/user/0
overlay          59G   53G  3.1G  95% /var/lib/docker/overlay/83b3f48705cf143225818f2cb8ef6a5e5a634a0915a719ee1c5cf8ed42212ff3/merged
overlay          59G   53G  3.1G  95% /var/lib/docker/overlay/2499db1fbf395c19d7de98a417ead5c9002d65ff58f6de14df5bc21506e68129/merged
shm              64M     0   64M   0% /var/lib/docker/containers/40c09a988385341d893fc19a02c49959023a992e5463d12a4b2c2c692b0b080a/shm
shm              64M     0   64M   0% /var/lib/docker/containers/78ec95fecfb21b415fa46161a4ddd9465f4309d906789714e9095e11ac2d6445/shm
overlay          59G   53G  3.1G  95% /var/lib/docker/overlay/d48c155765212df5b4215c555b09e3e2c1a32869b0c628b4b5bdf829d34022e0/merged
shm              64M     0   64M   0% /var/lib/docker/containers/d00ffe0dbcebaf176d60000c7ed91135ace7026b49c2903abf7f7f5b6317e720/shm
overlay          59G   53G  3.1G  95% /var/lib/docker/overlay/54ff581382a6c6c58c7ffe40b9351aaaadbbd5b31acc9487446b3ab5c8bd6fbd/merged
shm              64M     0   64M   0% /var/lib/docker/containers/abc3ff5e5d8f3e60346f5187c464d3148cefce3604fe0716ca27b8d4cd03cb5f/shm
overlay          59G   53G  3.1G  95% /var/lib/docker/overlay/a66aa00a04f0d22a4f63c8bb1f168cf0e0c772a7ac5adeee4b46f4dbc52265b1/merged
shm              64M     0   64M   0% /var/lib/docker/containers/d524c1a653b9e2681738ca1cf5312b58ec31ca0ac2f6c9600081214f2e916e6b/shm

Here you can see that 53G is used by overlay. So I do following:

[root@m13 lib]# du -sh /var/lib/docker
33G     /var/lib/docker

Where is additional 20G of space? Or maybe I missed something?

[root@m13 docker]# docker version
Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:23:59 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:23:59 2016
 OS/Arch:      linux/amd64

[root@m13 docker]# docker info
Containers: 5
 Running: 5
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 1.12.5
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-229.14.1.el7.centos.plus.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.856 GiB
Name: m13
ID: SMZF:64TM:BGWW:NU6U:M6TK:43F3:DHWW:RTFU:ZWIL:WDVV:MBRO:QAQQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

[gianarb]

It's easy for a registry to become so big. Just a question. Are you sharing a volume or mounting a directory into the registry that it's not into the Docker root dir? Where is your registry storing images?
Thanks

[euqen]

@gianarb I'm not sure, but I'm not sharing volume for registry. Here is what I did:

docker run -d -p 5000:5000 --restart=always --name registry \
  -v `pwd`/auth:/auth \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  -e REGISTRY_STORAGE_DELETE_ENABLED=true \
  registry

I'm not sure that it's registry's fault, I think it's can be something else ..

Also I forgot to say, that two containers are running in previleged mode, just to allow run docker into these containers. And yes, I checked these containers: there are no unused images or containers inside.

[cpuguy83]

overlay is reporting the space used on /
Notice how each overlay entry is reporting the same usage, which is also the same usage as /.
Presumably if du /var/lib/docker reports 33G, the rest is being used elsewhere.

Also note that the overlay graphdriver uses an excessive amount of inodes, which would also cause the system report "no space left on device" when inodes are exhausted.

[euqen]

Guys, this issue was resolved, the reason was that containers which are running in privileged mode has some unused volumes. So the following command helped me:
docker volume ls -qf dangling=true | xargs -r docker volume rm inside of each privileged container.

[ahamad37]

is this the same as docker volume prune ?
I have the same issue but when I removed dangling volumes, it only freed up few mega bytes. What would be the issue here?
When I do df -h I get a big list of 49GB overlay volumes