[Bug] Container Ephemeral storage limit is not enforced when using SOCI Snapshotter.

[debajyoti-truefoundry]

Description

This is what the resources section of my container looks like,

resources:
  limits:
    cpu: 50m
    ephemeral-storage: 20M
    memory: 50M
  requests:
    cpu: 20m
    ephemeral-storage: 10M
    memory: 20M

I exec"ed" in the container and executed the following commands.

/ # fallocate -l 1G test1
/ # fallocate -l 1G test2
/ # fallocate -l 1G test3
/ # du -sh .
3.0G    .
/ # fallocate -l 1G test4
/ # fallocate -l 1G test5
/ # du -sh .
5.0G    .
/ # 

Even after 5 minutes, the Pod did not get Evicted; the Container continued to run.

In Kubelet I can see logs like,

Feb 29 15:33:13 ip-10-2-30-208.eu-west-1.compute.internal kubelet[12965]: E0229 15:33:13.573821   12965 cri_stats_provider.go:448] "Failed to get the info of the filesystem with mountpoint" err="failed to get device for dir \"/var/lib/containerd/io.containerd.snapshotter.v1.soci\": stat failed on /var/lib/containerd/io.containerd.snapshotter.v1.soci with error: no such file or directory" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.soci"
Feb 29 15:33:23 ip-10-2-30-208.eu-west-1.compute.internal kubelet[12965]: E0229 15:33:23.574676   12965 cri_stats_provider.go:448] "Failed to get the info of the filesystem with mountpoint" err="failed to get device for dir \"/var/lib/containerd/io.containerd.snapshotter.v1.soci\": stat failed on /var/lib/containerd/io.containerd.snapshotter.v1.soci with error: no such file or directory" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.soci"

Steps to reproduce the bug

No response

Describe the results you expected

Ephemeral Storage Limits will be respected when using SOCI Snapshotter.

Host information

1. OS: AL2
2. Snapshotter Version: soci-snapshotter-grpc version v0.5.0 77f218d
3. Containerd Version: containerd github.com/containerd/containerd 1.7.11 64b8a811b07ba6288238eefc14d898ee0b5b99ba

Any additional context or information about the bug

No response

[Kern--]

This looks like a containers issue that already has a PR containerd/containerd#9216

I think that solution would also require a change to the containers config to tell it where SOCI is putting the snapshots. Maybe there’s something more we can do here.

[Kern--]

The upstream PR was closed because there was a different fix for containerd 2.0

I created a backport PR for 1.7 here: containerd/containerd#10096

There are slight containerd config changes needed to make it work, specifically specifying the snapshotter root dir:

[proxy_plugins.soci]
type = "snapshot"
address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"
[proxy_plugins.soci.exports]
root = "/var/lib/soci-snapshotter-grpc"

[Kern--]

This is fixed in containerd 1.7.16. It's merged into 1.6, but there hasn't been a release yet.

I'm going to mark this as fixed. Let me know if there are additional concerns.