[public repo] images should use high compression

[unclejack]

The images from the public repository should be using bzip2 -9 compression to speed up downloads and reduce traffic.

gzip compressed image:
pybuilder 288 MB

bzip2 -9 compressed image:
pybuilder.bz2 268 MB

savings: 20 MB, about 7%

[shykes]

While we're at it, we might as well enable high compression whenever docker creates a tarball. Eg. 'docker push' and 'docker export'.

[unclejack]

Using higher compression for all operations which create tarballs would be really nice.

The return is quite significant over time when it comes to the amount of storage needed on S3 and the network traffic.

I think only these changes are required to start using high compression for all new images on the registry:
registry.go:

 // FIXME: Don't do this :D. Check the S3 requierement and implement chunks of 5MB
 // FIXME2: I won't stress it enough, DON'T DO THIS! very high priority
- layerData2, err := Tar(path.Join(graph.Root, img.Id, "layer"), Gzip)
- layerData, err := Tar(path.Join(graph.Root, img.Id, "layer"), Gzip)
+ layerData2, err := Tar(path.Join(graph.Root, img.Id, "layer"), Bzip2)
+ layerData, err := Tar(path.Join(graph.Root, img.Id, "layer"), Bzip2)

[jpetazzo]

Don't use bzip2: use lzma/xz instead. xc is faster than bzip2 and achieves better compression; and it even has an "extreme" mode achieving even tighter compression (but then making it slower).

See e.g. http://pokecraft.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

Also, I couldn't check in the source, but we should obviously make sure that the layer hash is done on the uncompressed tar.

[shykes]

There is no layer hash right now, image IDs are computed randomly. I want
to bring back content-generated IDs and yes, that requires a tar-aware
checksum.

On Sunday, March 31, 2013, Jérôme Petazzoni wrote:

Don't use bzip2: use lzma/xz instead. xc is faster than bzip2 and
achieves better compression; and it even has an "extreme" mode achieving
even tighter compression (but then making it slower).

See e.g.
http://pokecraft.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO

Also, I couldn't check in the source, but we should obviously make sure
that the layer hash is done on the uncompressed tar.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/194#issuecomment-15697513
.

[unclejack]

I've done some tests with the pybuilder image:

468 MB pybuilder.tar
221 MB pybuilder.tar.gz - 47% of the original size
208 MB pybuilder.tar.bz2 - 44% of the original size
180 MB pybuilder.tar.xz - 38% of the original size

The xz lzma2 compressed image is 14% smaller than the bzip2 compressed image.

Other images show a similar decrease in size. Some even go down to 30% of the original size.

[jpetazzo]

Note: we would also have to update docker dependencies and installation instructions to tell people to install xz.

Bonus points if docker makes sure that xz is installed when starting, to get an informative error message rather than a cabalistic tar error (should that be another issue?)

[shykes]

The extra dependency is definitely a -1. Is it really worth the trouble compared to bzip2 -9?

[unclejack]

xz isn't required. bsdtar has native support for xz compression and it doesn't need xz from xz-utils, nor anything else.

I've just verified this by using bsdtar to compress in xz format, ran xz to make sure it's not there and then installed xz-utils to extract the archive. Everything worked.

So there's really nothing to warn about, other than about bsdtar's absence.

[shykes]

Awesome.

On Mon, Apr 1, 2013 at 10:26 AM, unclejack notifications@github.com wrote:

xz isn't required. bsdtar has native support for xz compression and it
doesn't need xz from xz-utils, nor anything else.

I've just verified this by using bsdtar to compress in xz format, ran xz
to make sure it's not there and then installed xz-utils to extract the
archive. Everything worked.

So there's really nothing to warn about, other than about bsdtar's absence.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/194#issuecomment-15725699
.

[titanous]

If we want to drop the bsdtar dependency we can. Just swap in archive/tar and http://godoc.org/code.google.com/p/lzma instead of shelling out. It may make sense to wait until the registry supports streaming upload to do this, but it's not required.

[shykes]

archive/tar doesn't support actual tarring and untarring on the filesystem.
Only parsing/encoding of the tar stream itself.

There is also auto-detection of compression which is a really useful
feature.

On Monday, April 1, 2013, Jonathan Rudenberg wrote:

If we want to drop the bsdtar dependency we can. Just swap in archive/tarhttp://tip.golang.org/pkg/archive/tar/and
http://godoc.org/code.google.com/p/lzma instead of shelling out. It may
make sense to wait until the registry supports streaming upload to do this,
but it's not required.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/194#issuecomment-15726255
.

[titanous]

@shykes Yeah, it would require essentially reimplementing the file walking and tar header building that tar/bsdtar does.

[unclejack]

The changes mentioned in this issue were made by pull request #308.

Another issue was created to add the hashing for layer contents and parent id when creating the image id. The issue is #310.