Device-mapper does not release free space from removed images

[AaronFriel]

Docker claims, via docker info to have freed space after an image is deleted, but the data file retains its former size and the sparse file allocated for the device-mapper storage backend file will continue to grow without bound as more extents are allocated.

I am using lxc-docker on Ubuntu 13.10:

Linux ergodev-zed 3.11.0-14-generic #21-Ubuntu SMP Tue Nov 12 17:04:55 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

This sequence of commands reveals the problem:

Doing a docker pull stackbrew/ubuntu:13.10 increased space usage reported docker info, before:

Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

And after docker pull stackbrew/ubuntu:13.10:

Containers: 0
Images: 3
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 413.1 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.8 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

And after docker rmi 8f71d74c8cfc, it returns:

Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-252:0-131308-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb
WARNING: No swap limit support

Only problem is, the data file has expanded to 414MiB (849016 512-byte sector blocks) per stat. Some of that space is properly reused after an image has been deleted, but the data file never shrinks. And under some mysterious condition (not yet able to reproduce) I have 291.5 MiB allocated that can't even be reused.

My dmsetup ls looks like this when there are 0 images installed:

# dmsetup ls
docker-252:0-131308-pool        (252:2)
ergodev--zed--vg-root   (252:0)
cryptswap       (252:1)

And a du of the data file shows this:

# du /var/lib/docker/devicemapper/devicemapper/data -h
656M    /var/lib/docker/devicemapper/devicemapper/data

How can I have docker reclaim space, and why doesn't docker automatically do this when images are removed?

[vreon]

+1, I'm very interested in hearing some discussion on this subject. My strategy so far has been

• be careful what you build/pull
• be prepared to blow away your /var/lib/docker 😐

@AaronFriel, which version of Docker are you on? 0.7.1?

[kiorky]

/cc @regilero (link also in to #2276)

[alexlarsson]

Starting from a fresh /var/lib/docker:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
292M -rw-------. 1 root root 100G Dec 12 17:29 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root   89 Dec 12 17:29 /var/lib/docker/devicemapper/devicemapper/json
732K -rw-------. 1 root root 2.0G Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

Then after docker pull busybox it grew a bit:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
297M -rw-------. 1 root root 100G Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root  181 Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/json
756K -rw-------. 1 root root 2.0G Dec 12 17:31 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 1
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 296.6 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

docker rmi busybox does not make the file larger, but makes the space free in the devicemapper pool:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
298M -rw-------. 1 root root 100G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root   89 Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/json
772K -rw-------. 1 root root 2.0G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 0
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 291.5 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

The loopback file doesn't grow if we download the image again:

# ls -lsh /var/lib/docker/devicemapper/devicemapper/*
298M -rw-------. 1 root root 100G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/data
4.0K -rw-------. 1 root root  181 Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/json
772K -rw-------. 1 root root 2.0G Dec 12 17:32 /var/lib/docker/devicemapper/devicemapper/metadata
# docker info
Containers: 0
Images: 1
Driver: devicemapper
 Pool Name: docker-0:31-15888696-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 296.6 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 0.7 Mb
 Metadata Space Total: 2048.0 Mb

So, it seems like we fail to re-sparsify the loopback file when the thinp device discards a block.

[alexlarsson]

However, if i create a file inside the container fs image it does reclaim the space in the loopback file.
I.e. I did this in the busybox image:

 cd lib
 cat libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 lib
c.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so
.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 libc.so.6 > a_file.bin
/lib # ls -l a_file.bin
-rw-r--r--    1 root     root      47090160 Dec 12 16:41 a_file.bin

This grew the data file from 299M to 344M. But when i removed a_file.bin (and waited a bit) it got back to 299M.

So, this seems to me like it is a devicemapper bug. I.e. it forwards discards from the thinp device to the underlying device, but it doesn't discard when removing thinp devices from the pool.

[alexlarsson]

This seem to be a kernel issue, I was looking at working around it bu using BLKDISCARD, but i failed. See this bugs for some details: https://bugzilla.redhat.com/show_bug.cgi?id=1043527

[alexlarsson]

I put my workaround in https://github.com/alexlarsson/docker/tree/blkdiscard, but we're still researching if we can do better than this.

[alexlarsson]

Upstream dm comments on this issue:
http://device-mapper.org/blog/2013/12/17/thin-provisioning-and-discard-support/

[blalor]

Having this problem on CentOS (2.6.32-358.23.2.el6.x86_64) with Docker 0.7.0, as well. Old, but the problem's not isolated to Ubuntu.

[skakri]

Same issue on Arch GNU/Linux 3.12.6-1-ARCH, Docker version 0.7.2.

[blalor]

Still exists on 0.7.0 on CentOS.

[nmz787]

Still exists in 0.7.2 on ubuntu 12.04.3 LTS.

A lot of the space is in docker/devicemapper/devicemapper/data and metadata, but also in docker/devicemapper/mnt

It's neat that I learned you can see the container file systems in docker/devicemapper/mnt/SOME_KIND_OF_ID/rootfs

but it's not neat that my hard disk is almost completely eaten up and only fixable by rmdir -r docker.

[logicminds]

I am having a similar issue while writing docker support for rspec-system. My test VM (docker host) has a 8GB drive and after repeatedly creating images without deleting them my drive fills up. But after removing all images and containers the drive is still 100% full. I figured it was an ID-10T error but just gave up and destroyed the VM all together.

[mengzechao]

Still exist in 0.7.5 on ubuntu 13.04.

[unclejack]

This issue has been fixed by PR #3256 which was recently merged. This fix will be included in a future release.

I'm closing this issue now because the fix has been merged to master.

[alexlarsson]

Note: Its not fully fixed until you also run a kernel with http://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=6d03f6ac888f2cfc9c840db0b965436d32f1816d in it. Without that the docker fix is only partial.

[logicminds]

What is the work around to remove space. I am using rhel 6.5 and it might be a while to get the new kernel.

Sent from my iPhone

On Jan 21, 2014, at 6:18 AM, Alexander Larsson notifications@github.com wrote:

Note: Its not fully fixed until you also run a kernel with http://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=6d03f6ac888f2cfc9c840db0b965436d32f1816d in it. Without that the docker fix is only partial.

—
Reply to this email directly or view it on GitHub.

[alexlarsson]

@logicminds There is no super easy way to recover the space atm. Technically it should be possible to manually re-sparsify the loopback files. But that would require all the non-used blocks to be zeroed or something for easily detection of the sparse areas, which is not done on thinp device removal.

[logicminds]

@alexlarsson Does this also affect OEL 6.5? The OEL 6.5 actually uses the uek 3.8 linux kernel and since I have the option between switching from 2.6 to 3.8 kernel this might be a simple switch for me.

[alexlarsson]

@logicminds I don't even know if that commit is in the upstream kernel yet. That link is from the device-mapper tree. Its definitely not in 3.8.

[alexlarsson]

I'm looking at creating a tool like fstrim that can be used to get back the space.

[blalor]

There's always rkt…

[ripienaar]

the general bitchy unhelpful snark is no doubt why no-one from upstream cares to give you proper answers.

[asarkar]

no one forced you to use Docker.

That's like Oracle telling a Java developer to use PHP due to a JVM bug. That's also not consistent with the elevator pitch here

Three years ago, Docker made an esoteric Linux kernel technology called containerization simple and accessible to everyone.

I'm sure a lot of people are grateful that Docker took off like it did and that couldn't have happened without volunteering from the community. However, it shouldn't be this hard to admit that it has it's problems too without implicitly dropping the "I'm a upstream contributor so shut up and listen" line whenever someone brings up an unlikable point.

[ror6ax]

Wait. I did report an issue, provided the details of my machine and setup,
which I'm not obliged to. None of devteam responded to my and others bug
reports in half a year period. Now I stated this fact, you call my behavior
bitchy? Do you even open-source? I'm looking for Go project to work on, and
it will not be Docker, I give you that. Is this your goal?
On 23 Jun 2016 16:45, "gregory grey" ror6ax@gmail.com wrote:

If this is not recommended feature to use - why is it silent and default
one? If you do not care for people using devicemapper - I might be even ok
with this. But do inform the user about it! Do you realize the amount of
headache people have due to this amazing 'workaround' you settled on??
On 23 Jun 2016 4:32 p.m., "kpande" notifications@github.com wrote:

workaround is to avoid using the docker device-mapper driver,
unfortunately.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#3182 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ACRlBbL5BDn3qMzUW_UiaALB32anUER4ks5qOpkJgaJpZM4BTlBd
.

[thaJeztah]

First of all, if you still have this problem, please open a new issue;

Wait. I did report an issue

You replied on a 3 year old, closed issue; following the discussion above, the original issue was resolved. Your issue may be the same, but needs more research to be sure; the errors you're reporing indicate that it may actually be something else.

I really recommend to open a new issue, not commenting on a closed issue

provided the details of my machine and setup, which I'm not obliged to.

You're not obliged to, but without any information to go on, it's unlikely to be resolved. So, when reporting a bug, please include the information that's asked for in the template:
https://raw.githubusercontent.com/docker/docker/master/.github/ISSUE_TEMPLATE.md

None of devteam responded to my and others bug reports in half a year period.

If you mean "one of the maintainers", please keep in mind that there's almost 24000 issues and PRs, and less than 20 maintainers, many of whom doing that besides their daily job. Not every comment will be noticed especially if it's on a closed issue.

If this is not recommended feature to use - why is it silent and default one?

It's the default if aufs, btrfs, and zfs are not supported, you can find the priority that's used when selecting drivers; see daemon/graphdriver/driver_linux.go. It's still above overlay, because unfortunately there's some remaining issues with that driver that some people may be affected by.

Automatically selecting a graphdriver is just to "get things running"; the best driver for your situation depends on your use-case. Docker cannot make that decision automatically, so this is up to the user to configure.

If you do not care for people using devicemapper - I might be even ok with this.

Reading back the discussion above, I see that the upstream devicemapper maintainers have looked into this multple times, trying to assist users reporting these issues, and resolving the issue. The issue was resolved for those that reported it, or in some cases, depended on distros updating devicemapper versions. I don't think that can be considered "not caring".

Also, why is default installation uses 'strongly discouraged' storage option?

Running on loop devices is fine for getting docker running, and currently the only way to set up devicemapper automatically. For production, and to get a better performance overall, use direct-lvm, as explained in the devicemapper section in the storage driver user guide.

Why wasn't I told so at installation?

That's out of scope for the installation, really. If you're going to use some software in production, it should be reasonable to assume that you get yourself familiar with that software, and know what's needed to set it up for your use case. Some maintainers even argued if the warning should be output at all. Linux is not a "holding hands" OS (does your distro show a warning that data loss can occur if you're using RAID-0? If you have ports opened in your Firewall?)

[shadowmint]

Deeply reluctant as I am, to once again resurrect this ancient thread, there is still no meaningful advice in it about how to work around this issue on an existing machine encountering this issue.

This is my best effort at a tldr; for the entire thread; I hope it helps others who find this thread.

Issue encountered

Your volume has a significant (and growing) amount of space which is in /var/lib/docker and you're using ext3.

Resolution

You're out of luck. Upgrade your file system or see blowing docker away at the bottom.

Issue encountered

Your volume has a significant (and growing) amount of space which is in /var/lib/docker and you're not using ext3 (eg. system currently using xfs or ext4)

Resolution

You may be able to reclaim space on your device using standard docker commands.

Read http://blog.yohanliyanage.com/2015/05/docker-clean-up-after-yourself/

Run these commands:

docker volume ls
docker ps
docker images

If you have nothing listed in any of these, see blowing docker away at the bottom.

If you see old stale images, unused containers, etc. you can perform manual cleanup with:

# Delete 'exited' containers
docker rm -v $(docker ps -a -q -f status=exited)

# Delete 'dangling' images
docker rmi $(docker images -f "dangling=true" -q)

# Delete 'dangling' volumes
docker volume rm $(docker volume ls -qf dangling=true)

This should reclaim much of the hidden container space in the devicemapper.

Blowing docker away

Didn't work? You're out of luck.

Your best bet at this point is:

service docker stop
rm -rf /var/lib/docker
service docker start

This will destroy all your docker images. Make sure to export ones you want to keep before doing this.

Ultimately, please read https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production; but I hope this will assist others who find this thread.

If you have problems with using advice above open a new ticket that specifically addresses the issue you encounter and link to this issue; do not post it here.

[gg7]

rm -rf /var/lib/docker

You can also use nuke-graph-directory.sh.

[gdring2]

After removing the files as stated above, I can't start Docker any more:
Mar 02 04:53:40 pmdc001b dockerd[29522]: Error starting daemon: error initializing graphdriver: open /var/lib/docker/devicemapper/devicemapper/data: no such file or directory

[pauvos]

Just ran into this issue on CentOS 7.3 and didn't want to debug devmapper issues that exist for more than 3 years, so I followed this DC/OS guide, purged the original packages, switched over to overlayfs and everything seems to work fine now: https://dcos.io/docs/1.7/administration/installing/custom/system-requirements/install-docker-centos/ (had to modify the ExecStart command for docker version 17.03 though -> "dockerd --storage-driver=overlay")

Server Version: 17.03.0-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
...
Operating System: CentOS Linux 7 (Core)

(purging volumes, images and containers didn't help. Deleting stuff in /var/lib/docker led to the problem described by @gdring2 )

[invisibleroads]

Running docker system prune freed a lot of space on my machines.

https://docs.docker.com/engine/reference/commandline/system_prune/

[Noah-Huppert]

Well, this is kind of... lame.

In my case I found this issue after I uninstalled Docker and deleted the the /var/lib/docker directory, so I couldn't run the equivalent of service docker stop ... service docker start.

I found that my system was not reporting the space from deleting /var/lib/docker as freed (I had ~14 GB sitting in what seemed like limbo).

The fix to this is to simply reload your file system, in my case I just rebooted and the space was reclaimed.

[shahaf600]

I can't believe it's still an issue! come on guys, i'm still having that

[thaJeztah]

@shahaf600 what version of docker are you running? Also see my comment above; #3182 (comment)

Without details there's not much to say about your situation; your case could be caused by a different issue, but resulting in a similar result.

[AndyFHAF]

good

[bghira]

after buying one of these pieces of garbage and seeing the state of support, i returned it.

[darkpixel]

There's your first problem @misterbigstuff...you bought something that's open source?

[bghira]

and returned it