Proposal: Image Acceleration(Apparate)

[kofj]

WIP. Update later.

Signed-off-by: fanjiankong fanjiankong@tencent.com

[tianon]

This proposal sounds conceptually very similar to https://github.com/containerd/stargz-snapshotter 🤔

[bergwolf]

And very even more similar to nydus image acceleration service: https://github.com/dragonflyoss/image-service

We've been discussing with Harbor team to create a pluggable image conversion mechanism that works for different image formats (currently nydus and estargz included). Maybe Apparate can join the force as well ;)

/cc @ktock

[xujihui1985]

what's the difference between https://github.com/dragonflyoss/image-service and this one?

[ktock]

We've been discussing with Harbor team to create a pluggable image conversion mechanism that works for different image formats (currently nydus and estargz included). Maybe Apparate can join the force as well ;)

👍

Recently a variety of image formats are discussed in the community (e.g. nydus, estargz, zstd:chunked...) not only Apparate, so it would be great to have a generic (and pluggable) conversion mechanism that works for them.

[imeoer]

A pluggable image conversion mechanism has also been proposed here: #167
We can participate in the discussion together. :)

[ghost]

It seems like another image-service (https://github.com/dragonflyoss/image-service), and another stargz (https://github.com/containerd/stargz-snapshotter).
I think Apparate, image-service and some other new image formats are based on or the extension of stargz for they look quite similar. It is better to make stargz as a standard and other implementations keep compatible with stargz and develop their own features.

[ktock]

@lovecontainers Standardization of lazy pulling in the current version of OCI Image Spec (v1) is discussed in opencontainers/image-spec#815.
nydus is proposed to the next version of OCI Image Spec (a.k.a. OCIv2). c.f. https://www.cncf.io/blog/2020/10/20/introducing-nydus-dragonfly-container-image-service/

[ghost]

@ktock yeah, I hope for the next oci spec. but nydus looks quite similar to stargz as it illustrated in the doc that nydus is a improvement of stargz. In fact, almost all newer remote image formats looks the same. So I think maybe is a better way to bring up the stargz v2 rather than so many stargz liked ones. At this moment, widely disscussion is necessary, but repeated ones are meaningless.

[lihuiba]

@lovecontainers Yes, repeated ones are meaningless. And there's a novel solution open-sourced recently:
https://github.com/alibaba/overlaybd
https://github.com/alibaba/accelerated-container-image
https://www.usenix.org/conference/atc20/presentation/li-huiba
https://www.usenix.org/conference/atc21/presentation/wang-ao

[ghost]

@lihuiba thank u, this is my first time learned about overlaybd for I am a beginner of containers. it looks like traditional vm image and native friendly to remote access. The most interesting point for me is your implementation deos not depends on FUSE.

[jiangliu]

It seems like another image-service (https://github.com/dragonflyoss/image-service), and another stargz (https://github.com/containerd/stargz-snapshotter).
I think Apparate, image-service and some other new image formats are based on or the extension of stargz for they look quite similar. It is better to make stargz as a standard and other implementations keep compatible with stargz and develop their own features.

There's a fundamental difference between stargz and nydus:)
Nydus could be thought as a file system over object storage and has a split fs metadata/data design, so different images could share data blob objects.

[kofj]

@malc0lm Pls subscribe and discuss here, we need to answer questions from the community.

[ghost]

It seems like another image-service (https://github.com/dragonflyoss/image-service), and another stargz (https://github.com/containerd/stargz-snapshotter).
I think Apparate, image-service and some other new image formats are based on or the extension of stargz for they look quite similar. It is better to make stargz as a standard and other implementations keep compatible with stargz and develop their own features.

There's a fundamental differen between stargz and nydus:)
Nydus could be thought as a file system over object storage and has a split fs metadata/data design, so different images could share data blob objects.

it is really a great improvement. it is hard to say a fundamental difference, and also the Apparatus. these similar propsosals may have competitions for business, for they stand for different companies, but make no sense for community reaching an agreement of next oci .

[lihuiba]

@lovecontainers @tianon Overlaybd is a combination of container image and VM image. It is a layered image in form of block device. It doesn't depends on FUSE / virtio-fs. I believe this design gathers the best of both worlds (container and VM), and it is applicable to both worlds.

[xujihui1985]

@lovecontainers @tianon Overlaybd is a combination of container image and VM image. It is a layered image in form of block device. It doesn't depends on FUSE / virtio-fs. I believe this design gathers the best of both worlds (container and VM), and it is applicable to both worlds.

interesting，good for you, you are so funny

[xujihui1985]

@lovecontainers @tianon Overlaybd is a combination of container image and VM image. It is a layered image in form of block device. It doesn't depends on FUSE / virtio-fs. I believe this design gathers the best of both worlds (container and VM), and it is applicable to both worlds.

obviously filesystem has higher abstract level than block device, which means, more business value can be added on top of it, and I don’t understand what makes you think this overlaybd thing is best of the world because it is not depend on fuse and virtiofs. But you depend on TCM which is another ko, so what is the advantage? You are welcome if you identify the pros and cons of different approach, instead you keep saying you are the best and others are meaningless which make me feel disgusting.

[lihuiba]

@xujihui1985 Hi, jihui. "It doesn't depends on FUSE / virtio-fs" is just a statement of fact, and a confirmation to lovecontainers. The reasons why I believe overlaybd is the best is complicated, and I suggest you read the papers above mentioned. There are paragraphs discussing this topic. Thanks!

[lihuiba]

@xujihui1985 Higher abstraction level doesn't necessarily mean better solution. For example, Python is a higher-level language than Java or C/C++, but Python is not necessarily better in every aspect. The best (-fit) abstractions vary in difference scenarios. The abstraction of block device doesn't preclude a file system abstraction on top of it. Actually, we have made an internal solution that includes an enhanced file system, called rofs, atop overlaybd. This solution unleashes all the imaginations about the file system abstraction, while retaining the advantages of block device, i.g. simplicity and efficiency.

[xujihui1985]

@xujihui1985 Higher abstraction level doesn't necessarily mean better solution. For example, Python is a higher-level language than Java or C/C++, but Python is not necessarily better in every aspect. The best (-fit) abstractions vary in difference scenarios. The abstraction of block device doesn't preclude a file system abstraction on top of it. Actually, we have made an internal solution that includes an enhanced file system, called rofs, atop overlaybd. This solution unleashes all the imaginations about the file system abstraction, while retaining the advantages of block device, i.g. simplicity and efficiency.

@lihuiba I don't get this metaphor, what's the matter with python? 😂 and I'm pleased to know you are working on a solution of filesystem. welcome to join the force. :)

[ghost]

@lovecontainers @tianon Overlaybd is a combination of container image and VM image. It is a layered image in form of block device. It doesn't depends on FUSE / virtio-fs. I believe this design gathers the best of both worlds (container and VM), and it is applicable to both worlds.

obviously filesystem has higher abstract level than block device, which means, more business value can be added on top of it, and I don’t understand what makes you think this overlaybd thing is best of the world because it is not depend on fuse and virtiofs. But you depend on TCM which is another ko, so what is the advantage? You are welcome if you identify the pros and cons of different approach, instead you keep saying you are the best and others are meaningless which make me feel disgusting.

@xujihui1985 I did some research on the basis of stargz, and really felt the bottleneck of FUSE, in both performance and stability. Did FUSE have any alternatives? or does nydus has some improvements on that ( no related statement found in nydus docs)?

[lihuiba]

@lovecontainers My team is also trying to improve fuse's performance, and we have an up-coming paper on this topic: https://www.usenix.org/conference/atc21/presentation/hsu .

But there's one more thing to solve: failure recovery. If fuse server process crashes, or gets killed, the file system instance may not recovery.

These problems (perforamce, fault-tolerance, etc.) do not exist in overlaybd.

[jiangliu]

@lovecontainers @tianon Overlaybd is a combination of container image and VM image. It is a layered image in form of block device. It doesn't depends on FUSE / virtio-fs. I believe this design gathers the best of both worlds (container and VM), and it is applicable to both worlds.

obviously filesystem has higher abstract level than block device, which means, more business value can be added on top of it, and I don’t understand what makes you think this overlaybd thing is best of the world because it is not depend on fuse and virtiofs. But you depend on TCM which is another ko, so what is the advantage? You are welcome if you identify the pros and cons of different approach, instead you keep saying you are the best and others are meaningless which make me feel disgusting.

@xujihui1985 I did some research on the basis of stargz, and really felt the bottleneck of FUSE, in both performance and stability. Did FUSE have any alternatives? or does nydus has some improvements on that ( no related statement found in nydus docs)?

At early stage of developing fs based image acceleration technologies, FUSE is a good choice. When the technology becomes mature, an in kernel read only fs may be better solution.
And nydus aims to become an in kernel fs:)

[xujihui1985]

@lovecontainers @tianon Overlaybd is a combination of container image and VM image. It is a layered image in form of block device. It doesn't depends on FUSE / virtio-fs. I believe this design gathers the best of both worlds (container and VM), and it is applicable to both worlds.

obviously filesystem has higher abstract level than block device, which means, more business value can be added on top of it, and I don’t understand what makes you think this overlaybd thing is best of the world because it is not depend on fuse and virtiofs. But you depend on TCM which is another ko, so what is the advantage? You are welcome if you identify the pros and cons of different approach, instead you keep saying you are the best and others are meaningless which make me feel disgusting.

@xujihui1985 I did some research on the basis of stargz, and really felt the bottleneck of FUSE, in both performance and stability. Did FUSE have any alternatives? or does nydus has some improvements on that ( no related statement found in nydus docs)?

@lovecontainers FUSE is not the problem of bottleneck, the problem is how to use fuse, the pros of stargz is the compatibility with targz, this is realy good one, the problem IMO is

1. each layers of stargz image will mount as a fuse mountpoint, and these layers then combine to overlayfs.
2. the toc must be fully load into rss for index inode, even the inode may never been read, that cause high memory footprint.

What nydus does to improve is to do "overlay" in build stage, and build the final view of root fs in metadata, so that one fuse mountpoint per image, underlying blob file is shared.
Instead of loading entire toc index into rss memory, nydus build a inode table in the header of metadata, so only a small portion of memory is needed during the startup, you can refer to the detailed design doc here https://github.com/dragonflyoss/image-service/blob/master/docs/nydus-design.md

[ghost]

@lovecontainers @tianon Overlaybd is a combination of container image and VM image. It is a layered image in form of block device. It doesn't depends on FUSE / virtio-fs. I believe this design gathers the best of both worlds (container and VM), and it is applicable to both worlds.

obviously filesystem has higher abstract level than block device, which means, more business value can be added on top of it, and I don’t understand what makes you think this overlaybd thing is best of the world because it is not depend on fuse and virtiofs. But you depend on TCM which is another ko, so what is the advantage? You are welcome if you identify the pros and cons of different approach, instead you keep saying you are the best and others are meaningless which make me feel disgusting.

@xujihui1985 I did some research on the basis of stargz, and really felt the bottleneck of FUSE, in both performance and stability. Did FUSE have any alternatives? or does nydus has some improvements on that ( no related statement found in nydus docs)?

@lovecontainers FUSE is not the problem of bottleneck, the problem is how to use fuse, the pros of stargz is the compatibility with targz, this is realy good one, the problem IMO is

1. each layers of stargz image will mount as a fuse mountpoint, and these layers then combine to overlayfs.
2. the toc must be fully load into rss for index inode, even the inode may never been read, that cause high memory footprint.

What nydus does to improve is to do "overlay" in build stage, and build the final view of root fs in metadata, so that one fuse mountpoint per image, underlying blob file is shared.
Instead of loading entire toc index into rss memory, nydus build a inode table in the header of metadata, so only a small portion of memory is needed during the startup, you can refer to the detailed design doc here https://github.com/dragonflyoss/image-service/blob/master/docs/nydus-design.md

yeah, I have already tried something similar to your solutions. thank you.

[ghost]

@kofj where is the git repository of Apparate? I am curious about Apparate's solution of recovering fuse process :)

[kofj]

An important goal of this proposal is to create a vendor-neutral sub-project in the goharbor community.

[malc0lm]

@lovecontainers Sorry, there is no Apparate repository in github currently. Recovering fuse process is core ability for Apparate. First, fuse in userspace and kernel fuse module use /dev/fuse fd to communitcate, so it must separate fuse process and holding fd process. And we also need fuse request tracing in case of io hang in recovering. Finally, in read/write fuse filesystem, we also need record opened fd.

[ghost]

@lovecontainers Sorry, there is no Apparate repository in github currently. Recovering fuse process is core ability for Apparate. First, fuse in userspace and kernel fuse module use /dev/fuse fd to communitcate, so it must separate fuse process and holding fd process. And we also need fuse request tracing in case of io hang in recovering. Finally, in read/write fuse filesystem, we also need record opened fd.

looking forward to see your implementation on github

[OrlinVasilev]

repo is here: https://github.com/goharbor/acceleration-service

[OrlinVasilev]

slack channel https://cloud-native.slack.com/archives/C01U31AK2LX

[jiangliu]

It seems like another image-service (https://github.com/dragonflyoss/image-service), and another stargz (https://github.com/containerd/stargz-snapshotter).
I think Apparate, image-service and some other new image formats are based on or the extension of stargz for they look quite similar. It is better to make stargz as a standard and other implementations keep compatible with stargz and develop their own features.

There's a fundamental difference between stargz and nydus:) Nydus could be thought as a file system over object storage and has a split fs metadata/data design, so different images could share data blob objects.

A status update about the nydus image service project(https://github.com/dragonflyoss/image-service). Recently we have published nydus v2.0, which includes an experiment rafsv6 image format. The rafsv6 image format is compatible with in the kernel EROFS filesystem, so a rafsv6 image could be directly mounted by the EROFS. And a patchset to integrate EROFS with fscache subsystem has been merged into linux 5.19-rc1.

With all this, an rafsv6 image could be used in two ways:

1. fuse + nydusd, for backward compatibility and extensible
2. erofs + fscache + nydusd, for high performance and reliability.

We are preparing articles to give more information about this topic too.

Thanks!

[jiangliu]

FYI：with the latest linux 5.19-rc1, nydus image could be mounted by the in-kernel EROFS:) https://d7y.io/blog/2022/06/06/evolution-of-nydus/

[OrlinVasilev]

@kofj can you check https://github.com/goharbor/acceleration-service and take decision to close this or rework it! Thank you!

[OrlinVasilev]

#167