Invoke Local for Python does cannot find handler module

[gmetzker]

This is a (Bug Report)

Description

For bug reports:

• What went wrong?

Using serverless v1.6.0 I'm trying to run a python lambda function locally but I receive an error that my handler module could not be found.

I was anticipating this would work since this feature was implemented in #2862

I suspect this might cause by my folder structure. In my case the python code is stored in src/serverless_lambda.py, I use an activated virtual environment, and a setup.py.

The setup.py is setup such that python can find modules in the src/* folder but serverless invoke local can't seem to find the package the same way python/pip can.

(venv) $:serverless-lambda-template gmetzker$ serverless invoke local --function serverlessLambda --data '{"command": "joke"}'
Traceback (most recent call last):
  File "/Users/gmetzker/hli-git/serverless-lambda-template/node_modules/serverless/lib/plugins/aws/invokeLocal/invoke.py", line 56, in <modul
e>

    module = import_module(args.handler_path.replace('/', '.'))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named serverless_lambda

This is how I create the setup of my project:

-- src
    -- __init__.py
    -- serverless_lambda.py
-- serverless.yml
-- setup.py
-- venv

1. Create virtual environment

virtualenv venv

2. Activate virtual environment

source ./venv/bin/activate

3. Create setup.py in root directory to look for modules in /src/serverless_lambda

from setuptools import Command, find_packages, setup


setup(
    name='serverless-lambda',
    version="1.0.0",
    author = 'Gary Metzker',
    py_modules='src/serverless_lambda',
    install_requires=[
        'boto3',
        'requests'
    ],
    packages = find_packages('src'),
    package_dir={'':'src'},
    extras_require = {
        'test': ['coverage', 'pytest', 'pytest-cov', 'pytest-json', 'mock'],
    },
    entry_points='''
        [console_scripts]
        serverless-lambda=serverless_lambda:main
    '''

)

4. Install pip dependencies (sourced form setup.py)

pip install --editable .

5. Create serverless.yml file in the project root.

service: serverlessLambdaTemplate

custom:
  stage: ${opt:stage, self:provider.stage}
  lambdaBaseName: serverless-template-lambda

provider:
  name: aws
  runtime: python2.7
  stage: dev
  region: us-west-2  

package:
  # Use package.sh to create package.zip before deploying.
  artifact: package.zip


functions:
  serverlessLambda:
    name: serverless-template-lambda-${self:custom.stage}
    handler: serverless_lambda.entry_point

6. Attempt to run serverless locally:

serverless invoke local --function serverlessLambda --data '{"command": "joke"}'

The other thing to note, when I deploy my lambda I typically have a separate package.sh script that I use to create the artifact package.zip. In the script zip up all *.py files from src/* into the root of the zip. Additionally the virtual environment site-packages are added to the zip.

• What did you expect should have happened?
  I would expect that serverless could use the same package context that was specified in setup.py and registered in the virtual environment with pip install --editable ..

It should be able to find src/serverless_lambda.py

• What was the config you used?
  See above.

• What stacktrace or error message from your provider did you see?

Error

Traceback (most recent call last):
  File "/Users/gmetzker/hli-git/serverless-lambda-template/node_modules/serverless/lib/plugins/aws/invokeLocal/invoke.py", line 56, in <modul
e>

    module = import_module(args.handler_path.replace('/', '.'))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named serverless_lambda

For feature proposals:

• What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.

  • Run local for python lambda functions
  • Use python virtual environments for a cleaner development environment
  • Store source code in nested folder like /src/lambda_handler.py so that directory structure is cleaner and source code is separate.
  • Use lambda setup.py to cleanly install python dependencies and specify package & module locations.
  • Use separate package.sh script to bundle /src/*.py and virtual environment files into a lambda.zip. I do this separately for our CI/CD system.

• If there is additional config how would it look

Similar or dependent issues:

• Invoke Local: Python Support #2862

Additional Data

• Serverless Framework Version you're using:
  1.6.0

• Operating System:
  OSX

• Stack Trace:

• Provider Error messages:

[dschep]

For packaging for deployment you might want to checkout serverless-python-requirements I wrote it too, but still need to push a stable release of 2.0.0.

[pmuens]

Closing this issue since #3346 should've solved it (see: #3346 (comment)).

Feel free to re-open if this issue still persists!

[ademidun]

Not sure if this is related to the problem mentioned above but If anyone comes here through Google try this:

npm install serverless --save-dev;
./node_modules/serverless/bin/serverless invoke local --function transferContent --data '{"content_type":"scholarship"}'