Lambda size limit

[amitaymolko]

What can we do to handle libraries that are too big for lambda?
For example 'scipy' (164mb) and numpy (77mb) are almost at the limit of 250mb for a lambda zip

[dschep]

We (@unitedincome) are using both of those on a project with serverless but aren't using this plugin with it yet. We're using a custom solution and shipping it packaged with tar -cvJ (Xz compression). Getting scipy and numpy to work well with this plugin is on my list of things todo when I get time 😄 \

[amitaymolko]

Hi great job!

Well I think we should wrap the custom variables like so:

custom:
  python-requirements:
    dockerizePip: true
    zipImport: true

Also i see some issues:
Files:
requirements.txt
requirements.zip
test.py.txt

So requirements.txt contains numpy, but it doesn't appear in the .requirements.zip
I attached my test.py file, which doesn't work because it can't load numpy
Also I noticed that the sizes for the libs are much smaller then i thought,
even in my virtualenv site-packages the sizes are much smaller than I previously had,
so Im not sure why scipy previously appeared as (164mb) it is now (70mb) in the site-packages.
scipy also doesn't appear in the .requirements.zip

[dschep]

@amitaymolko, what platform are you running on? and are you using the dockerizePip option?

[amitaymolko]

Running on OSX El Capitan
I ran without the dockerizePip option.
I will test this some more when I get the chance.
I will also pull the last few changes

[dschep]

Reopening because I realized you can't zipimport shared objects, so my solution doesn't work with numpy or scipy 😢

ZIP import of dynamic modules (.pyd, .so) is disallowed.

https://docs.python.org/2/library/zipimport.html

[marcoromelli]

Really like this plugin!
Is there some plan to improve on package size? I tried to package a function depending on numpy and tensorflow (with the dockerizePip option) but I get a .zip artifact of more than 60MB.

[dschep]

Did you use the zip option @marcoromelli? https://github.com/UnitedIncome/serverless-python-requirements#dealing-with-lambdas-size-limitations

[marcoromelli]

Actually you are right @dschep! Using the zip option the deploy works. The size of the artifact is still bigger than the 50MB limit, but for some reason AWS doesn't complain.

I still have a subtle problem related to this issue tensorflow/tensorflow#3086 which prevents the import of the Google Protobuf library.
The solution seems to be to create an empty __init__.py file in the google folder in the requirements but maybe there is a way to fix this in a more elegant way.

[dschep]

Ah, yes.. I've actually run into problems with namespace packages and pip install --target before on another project. I'm not sure what the best way to address that is. I'll create a separate ticket for that.