Creating scikit-learn Lambda Layer / Python 3.9 #27029
Unanswered
didomenicom
asked this question in
Q&A
Replies: 1 comment
-
I managed to make it work by building it on a docker container using the following repository: public.ecr.aws/lambda/python |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have been trying to build a lambda layer with a compressed version of scikit-learn for use in a model running on Lambda. Since Lambda's have a max size of 250 MB, I am using this script to strip out all of the files to make it as small as possible and utilize Lambda Layers.
The environment is setup to create the Lambda Layer via the AWS CDK (v2). To create the folder, I am using the following script. I have tried it on macOS and Gitlab Runner container (Python 3.9 alpine). Since I think there might be a compatibility issue with macOS, I am doing everything on linux.
The current error I am hitting is
On the line of my script
Do you have any suggestions on how I can get past this error?
Here is the script that builds the Lambda Layer (~175MB)
My requirements.txt file looks like
I have found reference to this error but I am using pip to install the package (and really trying to avoid building from source). One of the errors I did hit while trying to figure this out was
ModuleNotFoundError: No module named 'sklearn'
(reference) but has gone away once I added the layer folder to the path. This solution of installing scipy did not work either.I tried to use this as a reference (a bit dated) with the
--use-wheel --no-binary
options but they seem to be removed. I found reference to--platform
while trying to get psycopg2 added as a layer. From what I can tell themanylinux1_x86_64
doesn't exist for scikit-learn but there is amanylinux_2_17_x86_64
in the downloads and that at least doesn't error.Beta Was this translation helpful? Give feedback.
All reactions