EKS 1.26 + AL2 EKS AMI + SOCI...what am I doing wrong? 😬 #642

armenr · 2023-05-28T12:45:48Z

armenr
May 28, 2023

I've spent the weekend with SOCI. I feel like I'm super-close to getting this work, but I'm now stuck and uncertain as to what may be causing my issues.

If someone can please help me to get this working, you'd actually, literally, practically be changing the life (and the scaling constraints) of our Ops team. We're fighting with an image that's about 3 gigs in size...reducing startup time would be a huge win for our use-case. 🫠

Please excuse the verbosity of the my post, I just figured it's best to provide all the possible info that may be relevant.

Step 0 - Prep AL2 eks 1.26 image

Clone the amazon-eks-ami GitHub repo (AL2)
Modify install-worker.sh

Add this block @ line https://github.com/awslabs/amazon-eks-ami/blob/8369d06e5ae2a9c4c8cb99531ce525a767080019/scripts/install-worker.sh#L144

################################################################################
### SOCI #######################################################################
################################################################################

wget https://github.com/awslabs/soci-snapshotter/releases/download/v0.2.0/soci-snapshotter-0.2.0-linux-amd64.tar.gz
sudo tar -C /usr/local/bin -xvf soci-snapshotter-0.2.0-linux-amd64.tar.gz soci soci-snapshotter-grpc
rm soci-snapshotter-0.2.0-linux-amd64.tar.gz

sudo curl \
  --silent \
  --show-error \
  --retry 10 \
  --retry-delay 1 \
  -L "https://raw.githubusercontent.com/awslabs/soci-snapshotter/main/soci-snapshotter.service" \
  -o "/etc/systemd/system/soci-snapshotter.service"

sudo yum install fuse -y
sudo modprobe fuse
sudo bash -c 'echo "fuse" > /etc/modules-load.d/fuse.conf'

sudo systemctl daemon-reload
sudo systemctl enable --now soci-snapshotter

Bake AMI
Create EC2 from AMI and inspect:
i. ✅ check for soci and soci-snapshotter in shell/path
ii. ✅ check that the systemd unit for soci-snapshotter.service is doing what it's supposed to do
iii. ✅ check that fuse is doing what fuse is supposed to do

So far, so good!

Step 1 Generate SOCI index from existing ECR Image

I did all this from an EC2 that's identical to my EKS worker nodes (same OS, same IAM privileges and roles)

# Acquire ECR credentials

ecr_password=$(aws ecr get-login-password --region us-west-2)

# Pull existing image with explicit --platform

ctr images pull --platform linux/amd64 --user AWS:${ecr_password} REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:refactor-docker-workflow-consumeOneAndDie-rebased661-7.1.1-v3-0.1.5

# Modify image tag with unique "-soci" suffix

ctr i tag REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:refactor-docker-workflow-consumeOneAndDie-rebased661-7.1.1-v3-0.1.5 REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:refactor-docker-workflow-consumeOneAndDie-rebased661-soci

REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:refactor-docker-workflow-consumeOneAndDie-rebased661-soci

# Push newly-tagged image with explicit --platform
ctr i push --user AWS:${ecr_password} --platform linux/amd64 REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:refactor-docker-workflow-consumeOneAndDie-rebased661-soci

# Check ECR Repo for new image

aws ecr describe-images --repository-name redacted_repo --region us-west-2

        {
            "registryId": "REDACTED_ACCOUNT",
            "repositoryName": "redacted_repo",
            "imageDigest": "sha256:03dac96dbe0f8e422d3ce98dfb9cb7e70dd58b7e7762e76f65416913875345de",
            "imageTags": [
                "refactor-docker-workflow-consumeOneAndDie-rebased661-7.1.1-v3-0.1.4",
                "refactor-docker-workflow-consumeOneAndDie-rebased661-soci",
                "refactor-docker-workflow-consumeOneAndDie-rebased661-7.1.1-v3-0.1.5",
                "latest"
            ],
            "imageSizeInBytes": 3664174180,
            "imagePushedAt": 1685268526.0,
            "imageScanStatus": {
                "status": "FAILED",
                "description": "UnsupportedImageError: The operating system 'debian' version '12' is not supported."
            },
            "imageManifestMediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "artifactMediaType": "application/vnd.docker.container.image.v1+json"
        },

# Create SOCI index from newly-tagged image

soci create --platform linux/amd64 REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:refactor-docker-workflow-consumeOneAndDie-rebased661-soci
layer sha256:3a816d207b69abfb5e93b1560c436883c4e824f71447f63009791f540a6e34b2 -> ztoc skipped
layer sha256:4bf1b9fae878ee82f404aa205b6b57628e5e04be451cc24c334dcca18560ad83 -> ztoc skipped
layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 -> ztoc skipped
layer sha256:241328986e2c854dc2613b3690dcd6fb8e610f6a7a86f2759980594dd79e060c -> ztoc skipped
layer sha256:c60d3732da9dbf1cbd5b404b46f38a70b367fa89874f8ed877b8eb6f972ec983 -> ztoc skipped
layer sha256:69ce8ae71c323c539a148eeafaa5beb9047516497abc5864d6f724af24dcaea7 -> ztoc skipped
layer sha256:00896282c371e12bf68a71628b1941522018dc14cd6286054685ae570ed4b539 -> ztoc skipped
layer sha256:cce4507a6a9d6b55ef8c84e7e22f713e46e5b01e1abfa4c81266ca2d9c138502 -> ztoc skipped
layer sha256:55efc153a02896d60d9a6a8635117ebc65b5668163afa0cfcec4cd1866d65dcd -> ztoc skipped
layer sha256:f07ef98ef3ed283e97272846bb23ea7e7bdc905847a81006a99267e3a90d5313 -> ztoc skipped
layer sha256:841b7d674b150712d91b8b389194d9fadbdc7881ae88ac84d0b13aa2eabba4fb -> ztoc skipped
layer sha256:044c20bfa9859af550ff64d8beb3fbdaf67f0e7cc888aeaa417bffc23a11365a -> ztoc skipped
layer sha256:239de7db986790f3a3e09c421d0ea29c458f75a9676b9abb1ed2fb4ca22ad291 -> ztoc skipped
layer sha256:3be269d5483ff443e27aa648b614c45e3110687fe07778103378e281e89d07c0 -> ztoc skipped
layer sha256:10ed928730ab163bdcd852c62fe322c5da3259e7bbb2ceddf81ff3d5a6be2b47 -> ztoc skipped
layer sha256:1f722c915152960295221bd0dba3294b247abfba3c57021c386d21e599b5a8e2 -> ztoc skipped
layer sha256:da53d582a547dedcf11347a4cf23eb20ca8b925ebdb782779ff65cff5de6d1ea -> ztoc skipped
layer sha256:3f0b729d51496fe2b2a8b6029a605dc0acfaf74b934e64a6502fc4c3715081c5 -> ztoc skipped
layer sha256:d465868218c595310f1615cf7e7665112ce545bccb6589bc1a55fb0a491d445e -> ztoc skipped
layer sha256:8748a6db5202be1d88cd932ad6a6aec5248fef079de41b53af10326258bb8c1d -> ztoc skipped
layer sha256:1264fa8d0912fa1c505a7721fd62efc8c740e169d9e54f8686b7e503d32dbc26 -> ztoc skipped
layer sha256:9d2847ab7bac08aeafb9c1f722c9843562420e39a07a4b0e8d49005f6fcad5bc -> ztoc skipped
layer sha256:8dfc0f7f2f10fce0e2c2f4a987c8359ccc84583da188cf3909d48e50a853cb06 -> ztoc skipped
layer sha256:64c0aba11e30644c6e7618cc4f64dbaa0b8b0d2d8fc05e30653e60d297b63343 -> ztoc skipped
layer sha256:755b5a053bb56014fa3e997a58e26f56a2abd9684017f60a7b2d43a5eeb02d90 -> ztoc skipped
layer sha256:c921ba6dbbd52e8c153b2b7e308bf32a20c9b20b640346e1a6b01d979bb9edfe -> ztoc skipped
layer sha256:b86bb48a187a7eeb4db91d0c4f94a1f3fa99b08ca7692b0262c3b00bffe33d67 -> ztoc skipped
layer sha256:1fbf4ddd8ff268d6e24eb0ad1370691038c99f04b67821ea2dd029babd9e8ee7 -> ztoc skipped
layer sha256:36d21fdb44bafed912e5ca5ad956d49a90e34bfecc44f851c123b445cb49ff24 -> ztoc skipped
layer sha256:4b9df60e9a0f8e489771964e5ececeeb8930f1a4931f209dfff635850739c544 -> ztoc skipped
layer sha256:c8859e87640e18a1ae16736a885d08362da0aa7dfee7f7a218f3d99fe2856fe0 -> ztoc skipped
layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 -> ztoc skipped
layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 -> ztoc skipped
layer sha256:e56d1d92d71c495b685f02868f1fa184ee4daa4818b55b3d686be2ee72be1fee -> ztoc skipped
layer sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 -> ztoc skipped
layer sha256:f828e2168edfdd677f897a37cd182fd0ac0de032c7405b5ddc4f580e16cd88a6 -> ztoc skipped
layer sha256:9a93332df609cdff94a057581b00101c3a63aa112c90de52feb6082d249a6672 -> ztoc skipped
layer sha256:a23280f748ca2456ac380b28611a49301620cb0df6f13ff20ae1c60bdf3997e1 -> ztoc sha256:05c015ece4018a86881c0ed9852ebf5e04a72873021b86225212e157aeac402d
layer sha256:551a84846c2f95c5397350c2fcd99451582442fe0989fee7fe6aa9f97985f706 -> ztoc sha256:02ac45736b1f46b4963aa1238b332f1c2bbcc2cb155fe87b6d932b4d1ab8db40
layer sha256:9e3ea8720c6de96cc9ad544dddc695a3ab73f5581c5d954e0504cc4f80fb5e5c -> ztoc sha256:221e3496905f94c7d38b2ac38d482f11ca379c583f2ffc4a67a8543dccaa0a42
layer sha256:69c58f6e26f8e0a3cda8a2fed5bd7ffaa6108b85b8e3c7bd2d924d0fa8dfb8b0 -> ztoc sha256:f77153336d7791f6d3b235cd59af53d64b803cdd005042258ca6a97a44819b4e
layer sha256:46c825d64994d08f607867941bbc76009fd1d53607f2dd2457738c5dcc9fbf09 -> ztoc sha256:926b2299978cc2d193e3be1714c6d46faa6ba6121cadafb4c2eb2a8b31566f79
layer sha256:fcaf21cdaae20deb1d6cd561566c5d337ab94555f9d5ecd09af4ef6ecb926928 -> ztoc sha256:427e786499816dc55fee16504d8af02f2e67c6aaff34c9be9ea23f01d78430c0
layer sha256:1c3b81e384b0b69d6494aa5cf15e60325b749c76c9cbea42d3b1ec2e9d841756 -> ztoc sha256:27aa43a84c10e70c2516a01240558ac97794b812c8887b39759c4f4897158f7c
layer sha256:b54c967b20265b63807db3d97d951cc67d891ad40f657ef7ca96fd061c6dadf1 -> ztoc sha256:f56c16348d6ed574c71d1e6aaea1d27e15618c6a13a4db5a844d3eb5237d28ad
layer sha256:e21d2bc8ee069df01e0932c026dd3e4ab25bca38827857e9c35cf57f566a744b -> ztoc sha256:f39b7e81e975aeda6854357aee25f6b80918f3ad0d6212352ab3ae09f236a2c7
layer sha256:599b7288f007a5b6700331903a910bb85cd198fa96633405b49710b877c51517 -> ztoc sha256:9b9ae55449e52997959877cdbe9a2d96c4874a38324688b00900fc139423af0a
layer sha256:4fae59da4f397d50b56099fda784b02e3d074f0e56457ed59631644cf5ead837 -> ztoc sha256:64e186c8f0ab058bd31b3b9e536afd56fbadb3fb9374058164385192d7467d4b
layer sha256:80294e411325abe00c263d6cc3cc77ec601f294c7f9daf9309fdeeea32e818d9 -> ztoc sha256:af49d109e50bf6e5fae337fbe883a470be9d7ebffc3baf681e550dfae2c27178

# Push SOCI index

soci push --user AWS:${ecr_password} --platform linux/amd64 --existing-index allow REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:refactor-docker-workflow-consumeOneAndDie-rebased661-soci
pushing soci index with digest: sha256:bb1e175e1c79727dcff1ef0984df1b55aae88dd8d57beb3ee664a5d3d474dde9
skipped artifact with digest: sha256:221e3496905f94c7d38b2ac38d482f11ca379c583f2ffc4a67a8543dccaa0a42
skipped artifact with digest: sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
skipped artifact with digest: sha256:02ac45736b1f46b4963aa1238b332f1c2bbcc2cb155fe87b6d932b4d1ab8db40
skipped artifact with digest: sha256:27aa43a84c10e70c2516a01240558ac97794b812c8887b39759c4f4897158f7c
skipped artifact with digest: sha256:03dac96dbe0f8e422d3ce98dfb9cb7e70dd58b7e7762e76f65416913875345de
skipped artifact with digest: sha256:05c015ece4018a86881c0ed9852ebf5e04a72873021b86225212e157aeac402d
skipped artifact with digest: sha256:f77153336d7791f6d3b235cd59af53d64b803cdd005042258ca6a97a44819b4e
skipped artifact with digest: sha256:f56c16348d6ed574c71d1e6aaea1d27e15618c6a13a4db5a844d3eb5237d28ad
skipped artifact with digest: sha256:f39b7e81e975aeda6854357aee25f6b80918f3ad0d6212352ab3ae09f236a2c7
skipped artifact with digest: sha256:64e186c8f0ab058bd31b3b9e536afd56fbadb3fb9374058164385192d7467d4b
skipped artifact with digest: sha256:9b9ae55449e52997959877cdbe9a2d96c4874a38324688b00900fc139423af0a
skipped artifact with digest: sha256:af49d109e50bf6e5fae337fbe883a470be9d7ebffc3baf681e550dfae2c27178
skipped artifact with digest: sha256:427e786499816dc55fee16504d8af02f2e67c6aaff34c9be9ea23f01d78430c0
skipped artifact with digest: sha256:926b2299978cc2d193e3be1714c6d46faa6ba6121cadafb4c2eb2a8b31566f79
pushing artifact with digest: sha256:bb1e175e1c79727dcff1ef0984df1b55aae88dd8d57beb3ee664a5d3d474dde9
successfully pushed artifact with digest: sha256:bb1e175e1c79727dcff1ef0984df1b55aae88dd8d57beb3ee664a5d3d474dde9

Check ECR Repo to see if any new artifacts landed there

The first Artifact that's got a tag --> Info looks like this:

The second Artifact that's untagged --> Info looks like this:

So far, I think it's okay!?

Step 2

Modify Karpenter AWSNodeTemplate -->

  amiSelector:
    aws-ids: my-custom-soci-enabled-AL2-EKS-1.26-AMI

So, at this point, I deploy the container image I've just done all this work on, and cross my fingers in the hopes that I see the container pull and start faster...but no dice. Without SOCI = ~58 seconds to pull, with SOCI = ~58 seconds.

I referenced the image by its standard tag, of course, as mentioned in the README/documentation.

So, no improvements, OR it's not working as expected/I did something wrong.

Per the docs, I don't know how to tell if it's even working...but I did try this on the new worker node that's running the modified AL2 AMI, where the container image pulled and the pod started (at the same speed as before).

[root@ip-172-17-20-71 ~]# mount | grep fuse
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)

...and the pain begins!

Attempt 1 --> Use "Image tag" for "Artifact type --> Image Index" (from the screenshot)

# from helm-generated manifest

image: REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256-03dac96dbe0f8e422d3ce98dfb9cb7e70dd58b7e7762e76f65416913875345de

Also tried:

image: REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:sha256-03dac96dbe0f8e422d3ce98dfb9cb7e70dd58b7e7762e76f65416913875345de

Error:

  Warning  InspectFailed  6s (x4 over 31s)  kubelet            Failed to apply default image tag "REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256-03dac96dbe0f8e422d3ce98dfb9cb7e70dd58b7e7762e76f65416913875345de": couldn't parse image reference "REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256-03dac96dbe0f8e422d3ce98dfb9cb7e70dd58b7e7762e76f65416913875345de": invalid reference format
  Warning  Failed         6s (x4 over 31s)  kubelet            Error: InvalidImageName

Cool.

Attempt 2 --> Use "Digest" for "Artifact type --> Image Index" (from the screenshot)

# from helm-generated manifest

image: REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256:6cf56bd4f7f6d1f699f739606f4377c3293769eb6ad2d5b513cd68474b3113f6

Error:

  Normal   Pulling    8s (x2 over 23s)  kubelet            Pulling image "REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256:6cf56bd4f7f6d1f699f739606f4377c3293769eb6ad2d5b513cd68474b3113f6"
  Warning  Failed     8s (x2 over 23s)  kubelet            Failed to pull image "REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256:6cf56bd4f7f6d1f699f739606f4377c3293769eb6ad2d5b513cd68474b3113f6": rpc error: code = NotFound desc = failed to pull and unpack image "REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256:6cf56bd4f7f6d1f699f739606f4377c3293769eb6ad2d5b513cd68474b3113f6": failed to unpack image on snapshotter overlayfs: no match for platform in manifest sha256:6cf56bd4f7f6d1f699f739606f4377c3293769eb6ad2d5b513cd68474b3113f6: not found
  Warning  Failed     8s (x2 over 23s)  kubelet            Error: ErrImagePull

Cool...

Attempt 3 --> Use "Digest" for "Artifact type --> Other" (from the screenshot)

# from helm-generated manifest

image: REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256:bb1e175e1c79727dcff1ef0984df1b55aae88dd8d57beb3ee664a5d3d474dde9

Error:

  Normal   Pulling    11s (x2 over 25s)  kubelet            Pulling image "REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256:bb1e175e1c79727dcff1ef0984df1b55aae88dd8d57beb3ee664a5d3d474dde9"
  Warning  Failed     11s (x2 over 25s)  kubelet            Failed to pull image "REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256:bb1e175e1c79727dcff1ef0984df1b55aae88dd8d57beb3ee664a5d3d474dde9": rpc error: code = Unknown desc = failed to pull and unpack image "REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo@sha256:bb1e175e1c79727dcff1ef0984df1b55aae88dd8d57beb3ee664a5d3d474dde9": failed to unpack image on snapshotter overlayfs: mismatched image rootfs and manifest layers
  Warning  Failed     11s (x2 over 25s)  kubelet            Error: ErrImagePull

And here, I'm stuck. I'm sure I'm doing something stupid/obvious, but I've been at this for hours, so I'm probably really myopic.

Any help our advice would be greatly appreciated. Thank you!

armenr · 2023-06-25T04:18:39Z

armenr
Jun 25, 2023
Author

No help and no feedback. Yet another one of those AWS things where the hype is published, but no one gives a sh*t.

I wasted days on this thing, at the recommendation of an AWS employee, and got back zero feedback for trying to be a useful early customer.

Cool. Never going to use this again.

0 replies

Kern-- · 2023-06-27T16:56:41Z

Kern--
Jun 27, 2023
Maintainer

Hi. I'm really sorry we didn't respond to this discussion earlier. It's not the way we want to run this project and we want to do better. Going forward we're planning on reviewing new discussions and issues regularly to make sure that nothing sits with no repsonse. Despite our mistakes, we do care about this project and the experience of the people using it.

There are several pieces to answer your question.

The elephant in the room is this bug that means that the SOCI snapshotter doesn't support k8s right now: [FEATURE] Don't require layer size annotations #536. Once that's fixed, we're tracking verifying support in [FEATURE] Document Kubernetes Support #566. K8s isn't our primary use-case, but we've been hearing interest from users lately so we're planning to increase the priority on these issues.

EDIT: As of July 17 2023, both issues are merged. There is not yet a release that contains them, but the SOCI snapshotter will work with k8s if you build it from source.

The next issue with your setup is that containerd and k8s need to be configured to use the soci-snapshotter. For containerd, we have some instructions here, but there's a little more with k8s. You will need to add 2 sections to /etc/containerd/config.toml. The first is to get containerd to know how to connect with the soci-snapshotter:

[proxy_plugins]
  [proxy_plugins.soci]
    type = "snapshot"
    address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"

The second is to get containerd to use the SOCI snapshotter and pass the right labels to enable lazy loading:

[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "soci"
disable_snapshot_annotations = false

If you only want some of your pods to use SOCI, you can configure a runtime class instead. Please note that this requires containerd 1.7.0+:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.soci-runtime]
  snapshotter = "soci"

Then you can update your podspec (copied from this documentation)

apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  runtimeClassName: soci-runtime
  # ...

The third piece is that you don't need to change your image ref. You should use REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:refactor-docker-workflow-consumeOneAndDie-rebased661-soci exactly as if you weren't using soci. The SOCI snapshotter will go to the registry and find the SOCI index for you.

We have documentation about what each of those artifacts are here but the simplified version is:
Image Index -> bookkeeping so we can map an image to a SOCI index
Other -> the soci index (ECR now shows this as Soci Index)

You will also need to configure credentials for the SOCI snapshotter. We inherited our credential config from stargz, so these instructions should work (replacing containerd-stargz-grpc with soci-snapshotter-grpc where appropriate): https://github.com/containerd/stargz-snapshotter/blob/main/docs/overview.md#authentication
One other thing that sometimes trips people up is that SOCI indices apply to image content, not tag. Even though you retagged your image, the content didn't change. The SOCI index would work whether you pulled REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:refactor-docker-workflow-consumeOneAndDie-rebased661-soci or REDACTED_ACCOUNT.dkr.ecr.us-west-2.amazonaws.com/redacted_repo:refactor-docker-workflow-consumeOneAndDie-rebased661-7.1.1-v3-0.1.5

17 replies

chenxqdu Sep 15, 2023

hello Kern, I install soci v0.4.0 on one of my eks nodes, followed step2 to add [proxy_plugins] and [plugins."io.containerd.grpc.v1.cri".containerd] parts, but kubelet restart failed after I configured CRI-based authentication as mentioned https://github.com/containerd/stargz-snapshotter/blob/main/docs/overview.md#authentication, anything can be wrong ?

ducminhle Sep 19, 2023

Hi @chenxqdu,
It will be easier if you can provide log (ex: syslog, ...).
I use these command to login ECR and create cron:

cd /tmp
wget https://github.com/containerd/nerdctl/releases/download/v1.4.0/nerdctl-1.4.0-linux-amd64.tar.gz
tar -xzvf nerdctl-1.4.0-linux-amd64.tar.gz && rm -f nerdctl-1.4.0-linux-amd64.tar.gz
mv nerdctl /usr/local/bin/
aws ecr get-login-password --region <region> | /usr/local/bin/nerdctl login --username AWS --password-stdin <account id>.dkr.ecr.<region>.amazonaws.com
echo "aws ecr get-login-password --region <region> | /usr/local/bin/nerdctl login --username AWS --password-stdin <account id>.dkr.ecr.<region>.amazonaws.com" >> /etc/cron.hourly/ecrlogin
chmod +x /etc/cron.hourly/ecrlogin

and my container config:

cat /etc/containerd/config.toml 
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
snapshotter = "soci"
disable_snapshot_annotations = false
default_runtime_name = "runc"
discard_unpacked_layers = true

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause:3.5"

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"

[proxy_plugins]
 [proxy_plugins.soci]
   type = "snapshot"
   address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"

chenxqdu Sep 27, 2023

Thanks @ducminhle I was able to get soci working with the configuration you provided.
Here is the error log when I configured CRI-based authentication and restarted kubelet with "--image-service-endpoint=unix:///run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock":

Sep 18 00:12:28 ip-172-31-33-128.us-west-2.compute.internal kubelet[10649]: I0918 00:12:28.701315   10649 container_manager_linux.go:302] "Creating device plugin manager"
Sep 18 00:12:28 ip-172-31-33-128.us-west-2.compute.internal kubelet[10649]: I0918 00:12:28.701323   10649 manager.go:125] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 00:12:28 ip-172-31-33-128.us-west-2.compute.internal kubelet[10649]: I0918 00:12:28.701335   10649 server.go:66] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 00:12:28 ip-172-31-33-128.us-west-2.compute.internal kubelet[10649]: I0918 00:12:28.701355   10649 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 00:12:28 ip-172-31-33-128.us-west-2.compute.internal kubelet[10649]: I0918 00:12:28.703610   10649 remote_runtime.go:126] "Validated CRI v1 runtime API"
Sep 18 00:12:28 ip-172-31-33-128.us-west-2.compute.internal kubelet[10649]: E0918 00:12:28.705418   10649 run.go:74] "command failed" err="failed to run Kubelet: validate service connection: validate CRI v1 image API for endpoint \"unix:///run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
Sep 18 00:12:28 ip-172-31-33-128.us-west-2.compute.internal systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Sep 18 00:12:28 ip-172-31-33-128.us-west-2.compute.internal systemd[1]: Unit kubelet.service entered failed state.
Sep 18 00:12:28 ip-172-31-33-128.us-west-2.compute.internal systemd[1]: kubelet.service failed.

Kern-- Sep 27, 2023
Maintainer

Sorry about that. We didn't have an implementation for CRI v1 image service which is used by Kubernetes 1.26+ (see #810). While SOCI still works with Kubernetes 1.26+, it didn't work with the image-service credential mode (I should have caught this in my testing, but I tested with docker credentials instead of kubernetes credentails :( )

We have a PR to add this support in #843 which will be part of v0.5.0.

debajyoti-truefoundry Nov 30, 2023

We have a PR to add this support in #843 which will be part of v0.5.0.

@Kern-- Can you tell when will v0.5.0 be released?

gyanesh-mishra · 2023-10-30T01:12:51Z

gyanesh-mishra
Oct 30, 2023

@Kern-- Thank you and the team for making this awesome project.
I feel like I am on the same boat as the author being "so close" 🤏 to getting this working.

I currently see these errors

ct 30 01:08:07 ip-192-168-197-241.us-west-2.compute.internal soci-snapshotter-grpc[3414]: {"error":"cannot unpack the layer: cannot fetch layer: size of descriptoris 0; unable to resolve: unable to resolve ref (ACCOUNT_ID.dkr.ecr.us-west-2.amazonaws.com/IMAGE_NAME@sha256:ab16c7233ff6dff29659c50f1e588af7fae165332645294f2524e6af6d76282e): HEAD \"https://ACCOUNT_ID.dkr.ecr.us-west-2.amazonaws.com/v2/IMAGE_NAME/manifests/sha256:ab16c7233ff6dff29659c50f1e588af7fae165332645294f2524e6af6d76282e\": credential required for basic auth","key":"k8s.io/173/extract-755030004-_QVl sha256:fa5681f9df3b9f5d47f9675ead9a901b2ada8acdc07a27ca91e15387eaafa425","level":"warning","msg":"failed to prepare snapshot; deferring to container runtime","parent":"k8s.io/172/sha256:6af778dc49280d4fd326b6eaa919b69ebd00c7676d068d5b163675d1102ebe19","time":"2023-10-30T01:08:07.786698598Z"}

Even though my AWS EC2 instance role is authorized to access ECR.
If I do a docker pull, or nerdctl pull, the image pulls fine, but at no stage do I get the soci benefits since the snapshotter is failing to authenticate.

8 replies

gyanesh-mishra Nov 1, 2023

@Kern-- Since all commands run as root during instance startup, the ~ resolves to /root so that's definitely not the issue here.
I can validate by logging into the ec2 instance, switching to root and cat /root/.docker/config.json and see my contents in there

Kern-- Nov 1, 2023
Maintainer

Hmm. Sorry, I missed that.

The error message you added is pretty late in the process. Do you see any errors earlier?

Specifically, you should see some like Received status code: 401 Unauthorized. Refreshing creds... (this currently gets emitted once per layer, so you will likely see a lot). Anything after these?

failed to prepare remote snapshot logs may be useful, but these will probably also be missing auth if I had to guess.

You mentioned that the nerdctl command in your setup is working, so presumably it can access the docker creds so it's not a PATH issue, right?

gyanesh-mishra Nov 2, 2023

I did see quite a bit of Received status code: 401 Unauthorized. Refreshing creds... I didnt know that was normal 🤔
Let me scan the logs better.

Alternatively, if I do see output when I do mount | grep fuse, is that fair to say that things are working as intended?

Kern-- Nov 8, 2023
Maintainer

The 401 Unauthorized is expect - we try to make the request first then get creds if we get a 401. The error that you posted is not normal, but it sounds like there might be multiple things going on here. If you have more logs that you can share, that would be helpful.

mount | grep fuse should show as many fuse mounts as there are indexed layers in your SOCI index (by default we don't index layers < 10MiB). If it shows output then at least some of your layers are successfully lazy-loaded.

stanleyeze-upchain May 17, 2024

I am wondering if someone successfully got SOCI working on eks?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EKS 1.26 + AL2 EKS AMI + SOCI...what am I doing wrong? 😬 #642

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 25 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

EKS 1.26 + AL2 EKS AMI + SOCI...what am I doing wrong? 😬 #642

Step 0 - Prep AL2 eks 1.26 image

Step 1 Generate SOCI index from existing ECR Image

Step 2

Attempt 1 --> Use "Image tag" for "Artifact type --> Image Index" (from the screenshot)

Attempt 2 --> Use "Digest" for "Artifact type --> Image Index" (from the screenshot)

Attempt 3 --> Use "Digest" for "Artifact type --> Other" (from the screenshot)

Replies: 3 comments · 25 replies

armenr Jun 25, 2023 Author

Kern-- Jun 27, 2023 Maintainer

Kern-- Sep 27, 2023 Maintainer

Kern-- Nov 1, 2023 Maintainer

Kern-- Nov 8, 2023 Maintainer

Replies: 3 comments 25 replies

armenr
Jun 25, 2023
Author

Kern--
Jun 27, 2023
Maintainer

Kern-- Sep 27, 2023
Maintainer

Kern-- Nov 1, 2023
Maintainer

Kern-- Nov 8, 2023
Maintainer