[QUESTION] Image size limitations

[blankynap]

Hey SOCI Team,

great product!

i have a question (don't consider this one as a bug therefore want to ask first before creating a new issue): i've deployed the cfn-ecr-aws-soci-index-builder stack and want to try it out. The image size of my ML project according to ECR is 8633.48 Mb.
After i've pushed the image to ECR, the lambda cfn-ecr-aws-soci-index-bu-SociIndexGeneratorLambda was invoked and started producing logs as expected.

One of them is down below

{
    "level": "error",
    "error": "could not build zTOC for layer sha256:68e542c3e4792fe3787bf9ebea2c4673be4a581234cac1a50de257d55e977762: write /tmp/tmp.4113703201: no space left on device",
   
# lines ommited

    "message": "SOCI index build error"
}

I have checked the lambda config. The ephemeral storage size is set to 10240MB (max value).

Does it mean there is a size limitation of docker image and in my case i can not use the snapshotter because of the image size?

Thanks and regards!

[sparr]

There is a size limit for images that is lower than the available disk space in the environment where the index is being created. We are working on documenting this limitation and also improving the situation.

You can still use the snapshotter if you build your index in some way other than using the lambda, such as by running soci index in an environment where you have more disk space available and then pushing that index to your repository.

[blankynap]

@sparr thanks for the info! in my case the image is built and pushed to ECR by aws cdk during creation or update of the cloudforamation stack. so i technically can't pause the deployment to create a soci index and i also don't know the image tag in advance, which cdk will assign to the new image so i could build and tag the image locally beforehand.
since creating a soci make sense for the large images, don't you consider to replace the second lambda with e.g. a Fargate task to build an index, because Fargate can pull larger images up to 200 Gb (current ephemeral storage size)? Or as an alternative to connect the lambda to EFS.
meanwhile i will try to reduce the image size, although it's already optimized.

[sparr]

We are considering multiple possible solutions for situations such as this. Launching a container in an environment with higher storage caps is on the list of possibilities.

[blankynap]

sounds good! :) looking forward use this feature !

[blankynap]

@sparr

Update: i have moved the second lambda into a VPC and attached EFS to it (/tmp in a lambda source code i changed to /mnt/tmp as well), which solved the problem with the storage, but after that i got two more errors:

{
    "level": "error",
    "error": "unlinkat /mnt/tmp/d79c8a78-89f2-4b46-bed8-c1768f17b0cb2813747589/store/blobs/sha256: directory not empty",
    "ImageDigest": "sha256:8a7a7fdda843d3909daf072c5020412f17dd3fa4c65b98dce52281cbdcce5d64",
    "ImageTag": "1bb7565b8e2661da65cc58cdcdcaf4744c0df5d8d6408a3211e432f64d63f4e9",
    "RequestId": "d79c8a78-89f2-4b46-bed8-c1768f17b0cb",
    "time": "2023-08-15T12:56:21Z",
    "message": "Clean up error"
}

and

{
    "level": "error",
    "error": "Invocation timeout after 14 minutes and 50 seconds",
    "ImageDigest": "sha256:8a7a7fdda843d3909daf072c5020412f17dd3fa4c65b98dce52281cbdcce5d64",
    "ImageTag": "1bb7565b8e2661da65cc58cdcdcaf4744c0df5d8d6408a3211e432f64d63f4e9",
    "RequestId": "d79c8a78-89f2-4b46-bed8-c1768f17b0cb",
    "time": "2023-08-15T12:56:21Z",
    "message": "Invocation timeout error"
}

not sure about the reason for the first error, bit in case of the second one it looks like 15 min are not enough to create an index for image 8.5Gb in size... do you have any benchmark metrics?

One more thing... as i mentioned above i deploy with cdk and after image was built and pushed to ECR, it was immediately pulled by ECS, which of course don't wait until the index is built. This also would be great feature, because we don't build our images manually, but cdk deploy is a part of CICD.

[Kern--]

We don't have general benchmarks, but from what we've seen, there are issues somewhere in the 6-8GB range.

We had another user create a CDK construct that might work for you: #760 (comment)