orphaned diffs

[bukzor]

I'd like to know why docker uses so much disk, even after removing all containers, images, and volumes.
It looks like this "diff" has a layer, but the layer isn't referenced by anything at all.

/var/lib/docker/aufs/diff# du-summary
806628  c245c4c6d71ecdd834974e1e679506d33c4aac5f552cb4b28e727a596efc1695-removing
302312  a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
302304  957e78f9f9f4036689734df16dabccb98973e2c3de0863ef3f84de85dca8d92d
302256  8db1d610f3fbc71415f534a5d88318bbd2f3f783375813f2288d15f15846d312
288204  ac6b8ff4c0e7b91230ebf4c1caf16f06c1fdceff6111fd58f4ea50fc2dd5050b
288180  04a478c413ea80bcfa7f6560763beef991696eace2624254479e5e5dd69708c6
287804  d033ab6e4e5231dc46c6c417c680b239bb0e843024738517cbb0397128e166ca
233420  8e21143dca49e30cae7475b71b5aee9b92abe2069fbb9ab98ce9c334e3f6d4fa
212668  a631b94f7a2d5d21a96a78e9574d39cdeebbc81b51ac6c58bd48dc4045656477
205120  ae13341f8c08a925a95e5306ac039b0e0bbf000dda1a60afb3d15c838e43e349
205120  8d42279017d6095bab8d533ab0f1f7de229aa7483370ef53ead71fe5be3f1284
205116  59b3acd8e0cfd194d44313978d4b3769905cdb5204a590069c665423b10150e3
205116  040af0eee742ec9fb2dbeb32446ce44829cd72f02a2cf31283fcd067e73798ab
158024  ef0a29ff0b515c8c57fe78bcbd597243de9f7b274d9b212c774d91bd45a6c9b1
114588  061bd7e021afd4aaffa9fe6a6de491e10d8d37d9cbe7612138f58543e0985280
114576  149e8d2745f6684bc2106218711991449c452d4c7e6203e2a0f46651399162b0
114532  52b28112913abb0ed1b3267a0baa1cacd022ca6611812d0a8a428e61ec399589
114300  52475beba19687a886cba4bdb8508d5aaf051ceb52fb3a65294141ab846c8294
76668   4e6afb958b5ee6dea6d1a886d19fc9c780d4ecc4baeebfbde31f9bb97732d10d
76640   c61340c6a962ddd484512651046a676dbbc6a5d46aecc26995c49fe987bf9cdc

/var/lib/docker/aufs/diff# du -hs a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
296M    a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea

$ docker-find a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
+ docker=/var/lib/docker
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
/var/lib/docker/aufs/layers/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep -l a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
/var/lib/docker/aufs/layers/993e4988c510ec3ab4f6d139740a059df40585576f8196817e573a9684554c5c
/var/lib/docker/aufs/layers/95e68d59a8704f2bb52cc1306ca910ddb7af8956eb7c57970fcf7d8b3d9baddb
/var/lib/docker/aufs/layers/4e6afb958b5ee6dea6d1a886d19fc9c780d4ecc4baeebfbde31f9bb97732d10d
/var/lib/docker/aufs/layers/fd895b6f56aedf09c48dba97931a34cea863a21175450c31b6ceadde03f7b3da
/var/lib/docker/aufs/layers/ac6b8ff4c0e7b91230ebf4c1caf16f06c1fdceff6111fd58f4ea50fc2dd5050b
/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579-init
/var/lib/docker/aufs/layers/d5bbef5adf2efb6f15d4f96c4bee21beb955255d1ec17baf35de66e98e6c7328
/var/lib/docker/aufs/layers/9646360df378b88eae6f1d6288439eebd9647d5b9e8a471840d4a9d6ed5d92a4
/var/lib/docker/aufs/layers/cf9fd1c4a64baa39b6d6d9dac048ad2fff3c3fe13924b07377e767eed230ba9f
/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579
/var/lib/docker/aufs/layers/23ce5a473b101d85f0e9465debe5a0f3b8a2079b99528a797b02052d06bc11d8
/var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/cache-id

$ sudo cat /var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/diff
sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625

$ docker-find sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
+ docker=/var/lib/docker
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
+ sudo find /var/lib/docker '(' -path '/var/lib/docker/aufs/diff/*' -o -path '/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep -l sha256:b5185949ba02a6e065079660b0536672c9691fb0e0cb1fd912b2c7b29c91d625
/var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/diff

[bukzor]

# docker --version
Docker version 1.10.3, build 99b71ce

# docker info
Containers: 3
 Running: 0
 Paused: 0
 Stopped: 3
Images: 29
Server Version: 1.10.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 99
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 3.13.0-83-generic
Operating System: <unknown>
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 125.9 GiB
Name: dev34-devc
ID: VKMX:YMJ2:3NGV:5J6I:5RYM:AVBK:QPOZ:ODYE:VQ2D:AF2J:2LEM:TKTE
WARNING: No swap limit support

[bukzor]

I should also show that docker lists no containers, volumes, or images:

$ docker images -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

$ docker volume ls
DRIVER              VOLUME NAME

[thaJeztah]

strange; especially because of;

Containers: 3
 Running: 0
 Paused: 0
 Stopped: 3
Images: 29

which doesn't match the output of docker images / docker ps.

What operating system are you running on?

Operating System: <unknown>

@tonistiigi any idea?

[bukzor]

That was afterward. I guess some processes kicked off in the meantime.

The state I'm referring to (I have now) is:

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0

And I still have:

$ sudo du -hs /var/lib/docker/aufs/diff/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
296M    /var/lib/docker/aufs/diff/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea

[bukzor]

We're on Ubuntu Lucid with an upgraded kernel =/

$ uname -a
Linux dev34-devc 3.13.0-83-generic #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 10.04.1 LTS
Release:        10.04
Codename:       lucid

[HackToday]

It seems an interesting issue.
is it possible to have way to reproduce it ? @bukzor

[bukzor]

Surely it's possible, but I don't know how.
Please try running the below script on one of your active docker hosts and see what's left.
In our case, there's always plenty of diffs left behind.

#!/bin/bash
set -eu

echo "WARNING:: This will stop ALL docker processes and remove ALL docker images."
read -p "Continue (y/n)? "
if [ "$REPLY" != "y" ]; then
    echo "Aborting."
    exit 1
fi

xdocker() { exec xargs -P10 -r -n1 --verbose docker "$@"; }

set -x

# remove containers
docker ps -q | xdocker stop
docker ps -aq | xdocker rm

# remove tags
docker images | sed 1d | grep -v '^<none>' | col 1 2 | sed 's/ /:/' | xdocker rmi

# remove images
docker images -q | xdocker rmi
docker images -aq | xdocker rmi

# remove volumes
docker volume ls -q | xdocker volume rm

[tonistiigi]

One possible way I see this happening is that if there are errors on aufs unmounting. For example, if there are EBUSY errors then probably the image configuration has already been deleted before.

@bukzor Would be very interesting if there was a reproducer that would start from an empty graph directory, pull/run images and get it into a state where it doesn't fully clean up after running your script.

[bukzor]

That would be interesting, but sounds like a full day's work.
I can't commit to that.

Here's some more data regarding the (arbitrarily selected) troublesome diff above, a800.

$ docker-find a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea | sudo xargs -n1 wc -l | sort -rn
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep -l a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
15 /nail/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579
14 /nail/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579-init
13 /nail/var/lib/docker/aufs/layers/993e4988c510ec3ab4f6d139740a059df40585576f8196817e573a9684554c5c
12 /nail/var/lib/docker/aufs/layers/cf9fd1c4a64baa39b6d6d9dac048ad2fff3c3fe13924b07377e767eed230ba9f
11 /nail/var/lib/docker/aufs/layers/4e6afb958b5ee6dea6d1a886d19fc9c780d4ecc4baeebfbde31f9bb97732d10d
10 /nail/var/lib/docker/aufs/layers/23ce5a473b101d85f0e9465debe5a0f3b8a2079b99528a797b02052d06bc11d8
9 /nail/var/lib/docker/aufs/layers/95e68d59a8704f2bb52cc1306ca910ddb7af8956eb7c57970fcf7d8b3d9baddb
8 /nail/var/lib/docker/aufs/layers/ac6b8ff4c0e7b91230ebf4c1caf16f06c1fdceff6111fd58f4ea50fc2dd5050b
7 /nail/var/lib/docker/aufs/layers/fd895b6f56aedf09c48dba97931a34cea863a21175450c31b6ceadde03f7b3da
6 /nail/var/lib/docker/aufs/layers/d5bbef5adf2efb6f15d4f96c4bee21beb955255d1ec17baf35de66e98e6c7328
5 /nail/var/lib/docker/aufs/layers/9646360df378b88eae6f1d6288439eebd9647d5b9e8a471840d4a9d6ed5d92a4
4 /nail/var/lib/docker/aufs/layers/a8001a0e9515cbbda89a54120a89bfd9a3d0304c8d2812401aba33d22a2358ea
0 /nail/var/lib/docker/image/aufs/layerdb/sha256/d1c659b8e3d0e893e95c8eedc755adcb91a1c2022e1090376b451f7206f9b1c0/cache-id

So we see there's a chain of child layers, with f3286009193 as the tip.

$ docker-find f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579'$'
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep --color 'f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579$'
/nail/var/lib/docker/aufs/layers/f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep --color -l 'f3286009193f95ab95a16b2561331db06803ac536cea921d9aa64e1564046579$'
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/mount-id

So that layer was used in mount eb809c0321. I don't find any references to that mount anywhere:

$ docker-find eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -print
+ grep --color eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/mount-id
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/init-id
/nail/var/lib/docker/image/aufs/layerdb/mounts/eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e/parent
+ sudo find /nail/var/lib/docker '(' -path '/nail/var/lib/docker/aufs/diff/*' -o -path '/nail/var/lib/docker/aufs/mnt/*' ')' -prune -o -type f -print0
+ sudo xargs -0 -P20 grep --color -l eb809c0321a2501e61763333bc0dfb33ea0539c15957587f5de003ad21b8275e

Is there any way to find what container that mount was used for?
The doc only says the mount ID is no longer equal to the container ID, which isn't very helpful.
https://docs.docker.com/engine/userguide/storagedriver/aufs-driver/

[tonistiigi]

@bukzor eb809c0321 is the container id. What docs mean is that aufs id (f3286009193f in your case) is not container id.

/cc @dmcgowan as well

[bukzor]

@tonistiigi OK.

Then obviously the mount has outlived its container.

At what point in the container lifecycle is the mount cleaned up?
Is this the temporary writable aufs for running/stopped containers?

[tonistiigi]

@bukzor (rw) mount is deleted on container deletion. Unmount happens on container process stop. Diff folders are a place where the individual layer contents are stored, it doesn't matter is the layer is mounted or not.

[dmcgowan]

@bukzor The link between the aufs id and container id can be found at image/aufs/layerdb/mounts/<container-id>/mount-id. From just knowing an aufs id the easiest way to find the container id is to grep the image/aufs/layerdb directory for it. If nothing is found, then the cleanup was not completed cleanly.

[dennyzhang]

Running into similar issue.

We're running daily CI in the docker daemon server. /var/lib/docker/aufs/diff takes quite a mount of disk capacity, which it shouldn't be.

[jlukic]

Still 2gb in aufs/diff after trying everything reasonable suggested here or in related threads (including @bukzor's bash script above).

[mstorsjo]

Short of a proper fix, is there any straightforward way of removing the leftover mounts without removing all other images at the same time? (If no containers are running currently, I guess there should be no mounts, right?)

[newjanson]

I am experiencing the same issue. I am using this machine to test a lot of containers, then commit/delete. My /var/lib/docker/aufs directory is currently 7.9G heavy. I'm going to have to move this directory to another mount point, because storage on this one is limited. :(

[mcallaway]

# du -sh /var/lib/docker/aufs/diff/
1.9T    /var/lib/docker/aufs/diff/

[cpuguy83]

@mcallaway Everything in aufs/diff is going to be fs writes performed in a container.

[kayrus]

I have the same issue. All containers which I have are in running state, but there are lots of aufs diff directories which don't relate to these containers and relate to old removed containers. I can remove them manually, but it is not an option. There should be a reason for such a behavior.

I use k8s 1.3.5 and docker 1.12.

[kayrus]

Running of the docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc helped.

[garlab]

I have the same issue. I'm using Gitlab CI with dind (docker in docker).

[kayrus]

IMHO when image in the registry was updated within the same tag and it was pulled, then related container was restarted, old container and image are not GCed unless you run spotify/docker-gc.

Can someone else confirm this?

[thaJeztah]

@kayrus correct, docker will not automatically assume that an "untagged" image should also be removed. Containers could still be using that image, and you can still start new containers from that image (referencing it by its ID). You can remove "dangling" images using docker rmi $(docker images -qa -f dangling=true). Also, docker 1.13 will get data management commands (see #26108), which allow you to more easily cleanup unused images, containers, etc.

[kayrus]

@thaJeztah does /var/lib/docker/aufs/diff/ actually contain the "untagged" images?

[thaJeztah]

@kayrus yes they are part of the images (tagged, and untagged)

[carn1x]

getting a similar issue, no containers/images/volumes, ~13Gb of diffs

$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 1030
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-32-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.861 GiB
Name: gitrunner
ID: GSAW:6X5Z:SHHU:NZIM:O76D:P5OE:7OZG:UFGQ:BOAJ:HJFM:5G6W:5APP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

$ docker volume ls
DRIVER              VOLUME NAME
$ docker images -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
$

$ df -h
Filesystem                                 Size  Used Avail Use% Mounted on
...
/dev/mapper/gitrunner--docker-lib--docker   18G   15G  2.6G  85% /var/lib/docker

/var/lib/docker# sudo du -sm aufs/*
13782   aufs/diff
5       aufs/layers
5       aufs/mnt

[jadametz]

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: xfs
 Dirs: 1122

Same issue here. I understand 1.13 may get data management commands but in the meantime, I just want to safely delete the contents of this directory without killing Docker.

This is relatively blocking at this point.

[cpuguy83]

Yes, docker will not clean up old orphaned dirs yet.

[orf]

By old you mean those created from a previous version of docker with this issue?

This is a fresh install of Docker we did about two weeks ago, those orphans must have been created since then, so it seems that docker must still be creating those orphans?

I mean, in the last half an hour I've got 112 new diffs with -removing, since I rm'ed them manually.

$ ls /var/lib/docker/aufs/diff/ | grep removing | wc -l
112

You said "17.06 should no longer create orphaned diffs, but it won't clean up the old ones yet.", but surely this cannot be correct, or am I missing something? Are those tagged with -removing not orphaned?

[cpuguy83]

@orf On a newer kernel, it's highly unexpected to have any issue at all during removal. Are you mounting /var/lib/docker into a container?

I'll check in the aufs driver to see if there's a specific issue there with it reporting a successful remove when it really wasn't.

[orf]

We are not mounting /var/lib/docker into a container.

$ uname -a
Linux gitlab-cirunner 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

We are running 14.04 LTS

Let me know if there is anything I can provide to help debug this.

[rca]

For other reasons (swarm mode networking) I moved off 14.04 for Docker machines.

…

On Mon, Aug 21, 2017 at 8:23 AM orf ***@***.***> wrote: We are not mounting /var/lib/docker into a container. $ uname -a Linux gitlab-cirunner 4.4.0-83-generic #106~14.04.1-Ubuntu SMP Mon Jun 26 18:10:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux We are running 14.04 LTS — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#22207 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADRIfE2B2HNpbsKyTOj1CwGzulRT2C0ks5saaDPgaJpZM4IMGt2> .

[mmanderson]

This appears to be worse with 17.06.01-ce. I updated a build machine to this version and immediately started seeing the *-init-removing and the *-removing directories left around as part of the build process. I stopped the service, removed the /var/lib/docker directory, restarted the service and after a few builds was close to out of disk space again. I stopped the service again, ran apt-get purge docker-ce, removed /var/lib/docker again and installed the 17.06.0-ce version. Not getting the extra directories in /var/lib/docker/aufs/diff and disk space is representative of images that are on the build machine. I've reproduced the behavior on my development machine as well - just building an image seems to create these extra directories for each layer of the image so I would run out of disk space really quick. Again, reverting to 17.06.0-ce seems to not have the problem so I'm going to stay there for now.

[cpuguy83]

@mmanderson Thanks for reporting. Taking a look at changes in the AUFS driver.

[cpuguy83]

@mmanderson Do you have any containers in the Dead state in docker ps -a?

[blockjon]

All of my docker build servers are running out of space.

I have upgraded within the last week or so to Docker version 17.06.1-ce, build 874a737. I believe that nothing else has changed and that this issue either emerged or manifested as part of the upgrade process. The aufs diff directory is massive and I already pruned all images and dangling volumes.

[mmanderson]

issue-22207.txt
@cpuguy83 No containers in any state. Here is what I just barely did to demonstrate this with 17.06.01-ce:

1. Started with a fresh install of docker 17.06.01-ce on Ubuntu 16.04.03 LTS (i.e. docker not installed and no /var/lib/docker directory). After install verified an empty /var/lib/docker/aufs/diff directory.
2. Ran a docker build with a fairly simple dockerfile based on ubuntu:latest - all it does is pull statsd_exporter from github and extract it into /usr/bin (see attached file).
3. After running the build run docker ps -a to show no containers in any state. There are several *-remaining folder in the /var/lib/docker/aufs/diff folder.
4. Run docker system df to verify images, container, and volumes. Result is

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              2                   0                   132.7MB             132.7MB (100%)
Containers          0                   0                   0B                  0B
Local Volumes       0                   0                   0B                  0B

6. Running du -sch /var/lib/docker/*/ shows 152M for /var/lib/docker/aufs/
7. Run docker rmi $(docker images -q) to remove the built image layers. Running docker system df after this shows all zeros. Running du -sch /var/lib/docker/*/ shows 152M for /var/lib/docker/aufs/ and there are *-remaining folders for all of the folders that didn't have them before along with the existing *-remaining folders that are still there.

[rogaha]

@erikh is this the issue you are experiencing?

[mmanderson]

@cpuguy83 After uninstalling 17.06.01-ce, removing the /var/lib/docker directory and installing 17.06.0-ce I try to run the same build. The build fails because of the ADD from remote URL's bug that was fixed in 17.06.01. However I don't get any *-remaining directories for the steps that do complete and after cleaning up everything with docker system prune and docker rmi $(docker image -q) the /var/lib/docker/aufs/diff directory is again empty and the space is freed.

[cpuguy83]

Thanks all, this is a regression in 17.06.1...
PR to fix is here: #34587

[rogaha]

awesome, thanks for the quick patch @cpuguy83! /cc @erikh

[erikh]

@rogaha! yes, thanks to you and @cpuguy83!

[TP75]

Thank you so much @Karreg for your excellent script. After getting rid of all the old ophaned diffs and freeing huge amounts of lost disk space again we are using it now regularily to clean our VMs before installing new docker images. Great help and an almost perfect workaround for this issue now. @TP75

[osiyuk]

Looks like Docker, Inc. have some contracts with computer data storage manufacturers.

[mencos7]

@Karreg's script worked fine for me and I freed all the space in the diffs directory.

[kasunsjc]

Having the same issue.
Docker Host Details

root@UbuntuCont:~# docker info
Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 4
Server Version: 17.06.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 14
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-93-generic
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.358GiB
Name: UbuntuCont
ID: QQA5:DC5S:C2FL:LCC6:XY6E:V3FR:TRW3:VMOQ:QQKD:AP2M:H3JA:I6VX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

root@UbuntuCont:/var/lib/docker/aufs/diff# ls
031c85352fe85f07fede77dee0ac9dc2c7723177a819e72c534e1399208c95fa
09d53040e7e6798b5987ea76fe4f84f0906785b94a392a72e8e41a66cd9f242d
09d53040e7e6798b5987ea76fe4f84f0906785b94a392a72e8e41a66cd9f242d-init
0fb1ffc90969e9706801e2a18870f3ecd857a58f1094fbb968b3fa873e4cf2e4
10549179bd21a9c7af018d4ef305bb9196413b9662fce333b607104c40f38781
10d86a48e03cabf9af2c765dc84824809f24674ac339e4b9ffe572f50bd26b9c-init-removing
10d86a48e03cabf9af2c765dc84824809f24674ac339e4b9ffe572f50bd26b9c-removing
2e226946e8e6c2b3613de2afcff4cbb9890b6d9bd365fdda121a51ae96ec5606
2e226946e8e6c2b3613de2afcff4cbb9890b6d9bd365fdda121a51ae96ec5606-init
3601f6953132f557df8b52e03016db406168d3d6511d7ff5c08a90925ea288da-init-removing
3601f6953132f557df8b52e03016db406168d3d6511d7ff5c08a90925ea288da-removing
4b29141243aea4e70472f25a34a91267ab19c15071862c53e903b99740603d4c-init-removing
4b29141243aea4e70472f25a34a91267ab19c15071862c53e903b99740603d4c-removing
520e3fcf82e0fbbb48236dd99b6dee4c5bb9073d768511040c414f205c787dc5-init-removing
520e3fcf82e0fbbb48236dd99b6dee4c5bb9073d768511040c414f205c787dc5-removing
59cbb25a4858e7d3eb9146d64ff7602c9abc68509b8f2ccfe3be76681481904f
5d1c661b452efce22fe4e109fad7a672e755c64f538375fda21c23d49e2590f6
605893aba54feee92830d56b6ef1105a4d2166e71bd3b73a584b2afc83319591
63bd53412210f492d72999f9263a290dfee18310aa0494cb92e0d926d423e281-init-removing
63bd53412210f492d72999f9263a290dfee18310aa0494cb92e0d926d423e281-removing
72146e759ab65c835e214e99a2037f4b475902fdbe550c46ea0d396fb5ab2779-init-removing
72146e759ab65c835e214e99a2037f4b475902fdbe550c46ea0d396fb5ab2779-removing
8147e0b06dcbce4aa7eb86ed74f4ee8301e5fe2ee73c3a80dcb230bd0ddfcc26-init-removing
8147e0b06dcbce4aa7eb86ed74f4ee8301e5fe2ee73c3a80dcb230bd0ddfcc26-removing
a72735551217bb1ad01b77dbdbb9b8effa9f41315b0c481f8d74b5606c50deb4
aa58f2000b9f7d1ed2a6b476740c292c3c716e1d4dc04b7718580a490bba5ee8
b552cb853e33a8c758cb664aec70e2c4e85eacff180f56cbfab988a8e10c0174-removing
cd80c351b81ed13c4b64d9dfdc20c84f6b01cbb3e26f560faf2b63dae12dec55-init-removing
cd80c351b81ed13c4b64d9dfdc20c84f6b01cbb3e26f560faf2b63dae12dec55-removing
fe903be376821b7afee38a016f9765136ecb096c59178156299acb9f629061a2
fe903be376821b7afee38a016f9765136ecb096c59178156299acb9f629061a2-init

[cpuguy83]

@kasunsjc please read the posts just above yours.

[DBLaci]

I confirm upgrading to 17.06.2-ce solved this issue. I didn't have to manually the directories either (last time) after the upgrade.

[iay]

17.06.2-ce appears to have fixed this for me as well. No more -removing directories in there, got a decent amount of space back.

I'm assuming that the -init directories I have in aufs/diff are unrelated (some of them are pretty old). They are all small, though, so it hardly matters.

[C0rn3j]

Updating to 17.07.0 solved the issue here too, not even docker system prune --all -f would remove the directories before but after upgrading they got autoremoved on reboot.

[nocode99]

Confirming this issue was resolved on Ubuntu 16.04 with 17.06.2-ce. As soon as it was updated, the space cleared.

[thaJeztah]

Let me close this issue, as the original issue looks to be resolved