Add proposal for image conversion service #167

imeoer · 2021-06-03T08:29:13Z

Signed-off-by: Yan Song imeoer@linux.alibaba.com

bergwolf · 2021-06-03T08:53:33Z

@ktock Please see if the proposal makes sense for stargz

ktock

Thank you for this proposal!

ktock · 2021-06-03T09:03:44Z

proposals/new/image_conversion_service.md

+
+**Nydus:** Nydus image is a container accelerated image format provided by the Dragonfly image-service project, which offers the ability to pull image data on demand, without waiting for the entire image pull to complete and then start the container. It has been put in production usage and shown vast improvements over the old OCI image format in terms of container launching speed, image space, and network bandwidth efficiency, as well as data integrity. The Nydus image can also serve as an example and reference implementation for the on-going OCI image spec v2 discussion. For more details, please refer to [README](https://github.com/dragonflyoss/image-service).
+
+**Stargz:** Stargz is a lazily-pullable image format proposed by [stargz snapshotter project](https://github.com/containerd/stargz-snapshotter). Lazy pulling means a container can run without waiting for the pull completion of the image and necessary chunks of the image are fetched on-demand. This format is compatible to OCI/Docker images so this can be pushed to standard container registries (e.g. ghcr.io) as well as this is still runnable even on eStargz-agnostic runtimes including Docker. eStargz format is based on stargz image format by CRFS but comes with additional features like runtime optimization and content verification. For more details, please refer to [README](https://github.com/containerd/stargz-snapshotter).


Can we do s/stargz/estargz in this doc? Stargz Snapshotter project maintains eStargz but not stargz.

Thanks, fixed.

ktock · 2021-06-03T09:30:35Z

proposals/new/image_conversion_service.md

+```golang
+type Driver interface {
+  // Type returns the name of image format provider, e.g. nydus, stargz.
+  Type() string
+  // CheckHealth returns a nil error, means that the driver works fine.
+  CheckHealth() error
+  // Convert the source image to the target image, where Option provides
+  // some conversion options. If successful, the conversion result will
+  // be returned, otherwise a non-nil error will be returned.
+  Convert(ctx context.Context, sourceRef, targetRef string, opt Option) (Result, error)
+}
+```


According to this interface, pulling the original image and pushing the converted one is Driver's responsibility, right? If so, how are the registry credentials managed? (e.g. how does ICS provide the creds to the Driver)?

Thanks, neglected this part, how about this change?

With a webhook-based architecture, IMO ICS needs to fetch registry credential on its own instead of trying to get it from Harbor automatically. So maybe it can get it via k8s secrets etc. But I'm not sure how it works with Harbor user tenants. @steven-zou Do you have any suggestions?

ktock · 2021-06-03T09:41:11Z

proposals/new/image_conversion_service.md

+- Configure ICS to give robot permission and select a image conversion driver, such as Nydus or Stargz, then deploy as a webhook handle service.
+- Create a new http type webhook, set the endpoint url provided by ICS, allows to configure when to trigger image conversion based on event types (such as artifact push) and filter conditions (such as repository name).
+
+![Workflow](../images/image-conversion-service/workflow1.jpeg)


Could we have the definition of ~~Jov Service~~ Job Service mentioned in this figure?

Great, will do that. To my understanding, the Job Service in harbor is responsible for maintaining job queue and sending out the webhook requests, is that so? @wy65701436 .

ktock · 2021-06-03T11:23:46Z

proposals/new/image_conversion_service.md

+  // Base64 encoded <username:password> string.
+  Auth string


I think username and password should have separated fields from each other.

And how is tls-related configuration passed to the Driver?

I think passing the registry-related configuration to Driver will make the interface complex. So how about performing pull/push on ICS-side (not Driver)? Image contents can be shared between ICS and Driver using something like the content store of containerd.

Should be ok.

Does it refer to Internal TLS communication?

I agree that ICS should do the pull and push operations, not Driver itself. If using content store, will we have dependencies on containerd and snapshotter service? Nydusify (a standalone nydus image conversion tool) currently fetches the tar.gz file of layer directly and unpacks it into a directory to do the conversion. However, nydusify should really consider taking advantage of content store and snapshotter snapshots to eliminate the need to pull and unpack, which the nydus Driver can adapt. Maybe we can design like:

Convert(ctx context.Context, src ocispec.Descriptor, opt Option) (dst *ocispec.Descriptor, error)

Does it refer to Internal TLS communication?

According to the proposal ICS contains configurations related to cert, etc.

# https related config https: # https port for harbor, default is 443. port: 443 # The path of cert and key files. certificate: /path/to/certificate private_key: /path/to/private_key

So I was interested in how these information will be shared to Driver.
But now we are discussing about doing pull/push fully inside ICS (not Driver) so we don't need to care about it anymore.

Convert(ctx context.Context, src ocispec.Descriptor, opt Option) (dst *ocispec.Descriptor, error)

LGTM about this interface.

I think snapshotter and content store are separated components and don't depend on each other. So having the content store in ICS is just enough to share image contents with Driver.

But now we are discussing about doing pull/push fully inside ICS (not Driver) so we don't need to care about it anymore.

Ok, it makes sense.

I think snapshotter and content store are separated components and don't depend on each other. So having the content store in ICS is just enough to share image contents with Driver.

The nydus conversion relies on the unpacked directory tree, so a snapshotter may be needed for nydus conversion in the future, but in any case, I think this interface is feasible for now.

reasonerjt · 2021-06-04T07:20:20Z

Could you elaborate how does end user map the converted image to the original one?

I assume the converted image can't be recognized by the scanner, and cannot be signed? How do we ensure the project level policies are enforced to converted images such that vulnerable ones are blocked?

IMO, this is more important than inter-op between Harbor and the ICS.

imeoer · 2021-06-04T07:35:54Z

Could you elaborate how does end user map the converted image to the original one?

I assume the converted image can't be recognized by the scanner, and cannot be signed? How do we ensure the project level policies are enforced to converted images such that vulnerable ones are blocked?

IMO, this is more important than inter-op between Harbor and the ICS.

@reasonerjt We can configure the target_suffix or multi_platform option to specify what the reference of converted image will look like. The converted image is still OCI compatible and can be signed, but the scanner may not be able to unpack and scan the contents of converted image, should we assume that the converted image is trusted after the original image has been certified by the scanner?

reasonerjt · 2021-06-04T07:41:32Z

 We can configure the target_suffix or multi_platform option to specify what the reference of converted image will look like.

By reference do you mean tag?

should we assume that the converted image is trusted after the original image has been certified by the scanner?

I think if the relationship can be enforced then yes, using tag or something that can be easily modified is not very reliable.

imeoer · 2021-06-04T07:51:55Z

@reasonerjt Yes, means tag in target_suffix. how about merging the original image and the converted image into a manifest index like this.

reasonerjt · 2021-06-04T08:46:39Z

There may be race condition when 2 convertors try to modify the same index.json?
Or do you mean each convertor will create one separate index.json?
Since it's a converted image and it's an OCI artifact, how about adding it into manifest?

imeoer · 2021-06-04T08:57:38Z

@reasonerjt Maybe we can design a workflow to avoid the race condition situation, for example, we can put the conversion job into a FIFO queue by repo level, the previous job will be canceled once has a new job be submitted in the same repo.

In addition, an annotation can be added to the converted image to record the digest of original image, which the ICS or user can check to decide whether to push or use the converted image.

liubogithub · 2021-06-04T18:19:32Z

I think if the relationship can be enforced then yes, using tag or something that can be easily modified is not very reliable.

@reasonerjt
There is an on-going artifact pr 29 which adds a new artifact manifest, it represents two things, a) a new artifact, b) the relationship between the artifact and the existing image or artifact (via subjectManifest).

{
  "schemaVersion": 3,
  "mediaType": "application/vnd.oci.artifact.manifest.v1-rc1+json",
  "referenceType": "cncf.nydus.v1-rc1",
  "blobs": [
    {
      "mediaType": "application/vnd.cncf.nydus.bootstrap.v1.tar+gzip",
      "digest": "sha256:f6bb0822fe567c98959bb87aa316a565eb1ae059c46fa8bba65b573b4489b44d",
      "size": 32654
    },
    {
      "mediaType": "application/vnd.cncf.nydus.blob.v1",
      "digest": "sha256:d35808e58856ef91d07dedf94b93301b6efdfcc831477d3b1bb37e6c3e19cce6",
      "size": 25851449
    },
    {
      "mediaType": "application/vnd.cncf.nydus.blob.v1",
      "digest": "sha256:dbad66bcfe29ef383157a3e122acbd08cd2ebd40f5658afa2ae62c52ffe26e9f",
      "size": 226
    }
  ],
  "subjectManifest": {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:73c803930ea3ba1e54bc25c2bdc53edd0284c62ed651fe7b00369da519a3c333",
    "size": 16724
  },
  "annotations": {
    "org.cncf.nydus.v1.author": "wabbit-networks.io"
  }
}

imeoer · 2021-06-07T03:22:02Z

There is an on-going artifact pr 29 which adds a new artifact manifest, it represents two things, a) a new artifact, b) the relationship between the artifact and the existing image or artifact (via subjectManifest).

@liubogithub @reasonerjt This also looks good. How to avoid inconsistency between the original and converted image is a good question, and I've added it to the end of the proposal. It'd be a pleasure for us to discuss this in the image acceleration working group.

Signed-off-by: Yan Song <imeoer@linux.alibaba.com>

Remove the content about the trigger method in Harbor side. Signed-off-by: Yan Song <imeoer@linux.alibaba.com>

imeoer force-pushed the master branch from 0528623 to b0d9efc Compare June 3, 2021 08:42

imeoer mentioned this pull request Jun 3, 2021

Proposal: Image Acceleration(Apparate) #165

Open

ktock reviewed Jun 3, 2021

View reviewed changes

imeoer force-pushed the master branch 3 times, most recently from fb9589a to da9cb8c Compare June 3, 2021 09:58

ktock reviewed Jun 3, 2021

View reviewed changes

imeoer force-pushed the master branch from da9cb8c to 02be381 Compare June 4, 2021 07:17

imeoer force-pushed the master branch from 02be381 to f5cef2b Compare June 7, 2021 03:11

Add proposal for image conversion service

e9f2e1d

Signed-off-by: Yan Song <imeoer@linux.alibaba.com>

imeoer force-pushed the master branch from f5cef2b to e9f2e1d Compare June 7, 2021 03:24

steven-zou added the area/image-acceleration label Jun 10, 2021

Update proposal for image conversion service

bb5fdf8

Remove the content about the trigger method in Harbor side. Signed-off-by: Yan Song <imeoer@linux.alibaba.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add proposal for image conversion service #167

Add proposal for image conversion service #167

imeoer commented Jun 3, 2021

bergwolf commented Jun 3, 2021

ktock left a comment

ktock Jun 3, 2021

imeoer Jun 3, 2021

ktock Jun 3, 2021

imeoer Jun 3, 2021

bergwolf Jun 3, 2021

ktock Jun 3, 2021 •

edited

imeoer Jun 3, 2021

ktock Jun 3, 2021

imeoer Jun 4, 2021

ktock Jun 4, 2021

imeoer Jun 4, 2021 •

edited

reasonerjt commented Jun 4, 2021

imeoer commented Jun 4, 2021 •

edited

reasonerjt commented Jun 4, 2021 •

edited

imeoer commented Jun 4, 2021

reasonerjt commented Jun 4, 2021 •

edited

imeoer commented Jun 4, 2021 •

edited

liubogithub commented Jun 4, 2021 •

edited

imeoer commented Jun 7, 2021


		Nydus: Nydus image is a container accelerated image format provided by the Dragonfly image-service project, which offers the ability to pull image data on demand, without waiting for the entire image pull to complete and then start the container. It has been put in production usage and shown vast improvements over the old OCI image format in terms of container launching speed, image space, and network bandwidth efficiency, as well as data integrity. The Nydus image can also serve as an example and reference implementation for the on-going OCI image spec v2 discussion. For more details, please refer to [README](https://github.com/dragonflyoss/image-service).

		Stargz: Stargz is a lazily-pullable image format proposed by [stargz snapshotter project](https://github.com/containerd/stargz-snapshotter). Lazy pulling means a container can run without waiting for the pull completion of the image and necessary chunks of the image are fetched on-demand. This format is compatible to OCI/Docker images so this can be pushed to standard container registries (e.g. ghcr.io) as well as this is still runnable even on eStargz-agnostic runtimes including Docker. eStargz format is based on stargz image format by CRFS but comes with additional features like runtime optimization and content verification. For more details, please refer to [README](https://github.com/containerd/stargz-snapshotter).

Add proposal for image conversion service #167

Are you sure you want to change the base?

Add proposal for image conversion service #167

Conversation

imeoer commented Jun 3, 2021

bergwolf commented Jun 3, 2021

ktock left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ktock Jun 3, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imeoer Jun 4, 2021 • edited

Choose a reason for hiding this comment

reasonerjt commented Jun 4, 2021

imeoer commented Jun 4, 2021 • edited

reasonerjt commented Jun 4, 2021 • edited

imeoer commented Jun 4, 2021

reasonerjt commented Jun 4, 2021 • edited

imeoer commented Jun 4, 2021 • edited

liubogithub commented Jun 4, 2021 • edited

imeoer commented Jun 7, 2021

ktock Jun 3, 2021 •

edited

imeoer Jun 4, 2021 •

edited

imeoer commented Jun 4, 2021 •

edited

reasonerjt commented Jun 4, 2021 •

edited

reasonerjt commented Jun 4, 2021 •

edited

imeoer commented Jun 4, 2021 •

edited

liubogithub commented Jun 4, 2021 •

edited