'Importing the numpy c-extensions failed.' in AWS Lambda deployed to IoT GreenGrass with Pandas import

[OscarVanL]

I'm deploying an AWS Lambda Function to an edge device with IoT GreenGrass, when my function runs it has a problem with my import pandas requirement giving an error related to numpy.

See this thread on the AWS GreenGrass forums

Reproducing code example:

import pandas as pd

(Numpy is a dependency of pandas)

I've written this using the AWS Toolkit for PyCharm, inside that there's a requirements.txt file listing numpy and pandas. The SAM build process has been set to complete in a container so as to avoid using numpy/pandas compiled for Windows.

Error message:

Here's the log file:

[2019-09-17T11:20:04.778+01:00][FATAL]-lambda_runtime.py:131,Failed to import handler function "doGreenGrassPredictions.function_handler" due to exception: Unable to import required dependencies:
[2019-09-17T11:20:04.778+01:00][FATAL]-numpy:
[2019-09-17T11:20:04.778+01:00][FATAL]-IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
[2019-09-17T11:20:04.778+01:00][FATAL]-Importing the numpy c-extensions failed.
[2019-09-17T11:20:04.778+01:00][FATAL]-- Try uninstalling and reinstalling numpy.
[2019-09-17T11:20:04.778+01:00][FATAL]-- If you have already done that, then:
[2019-09-17T11:20:04.778+01:00][FATAL]-  1. Check that you expected to use Python3.7 from "/usr/bin/python3.7",
[2019-09-17T11:20:04.778+01:00][FATAL]-     and that you have no directories in your PATH or PYTHONPATH that can
[2019-09-17T11:20:04.778+01:00][FATAL]-     interfere with the Python and numpy version "1.17.2" you're trying to use.
[2019-09-17T11:20:04.778+01:00][FATAL]-  2. If (1) looks fine, you can open a new issue at
[2019-09-17T11:20:04.778+01:00][FATAL]-     https://github.com/numpy/numpy/issues.  Please include details on:
[2019-09-17T11:20:04.778+01:00][FATAL]-     - how you installed Python
[2019-09-17T11:20:04.778+01:00][FATAL]-     - how you installed numpy
[2019-09-17T11:20:04.778+01:00][FATAL]-     - your operating system
[2019-09-17T11:20:04.778+01:00][FATAL]-     - whether or not you have multiple versions of Python installed
[2019-09-17T11:20:04.778+01:00][FATAL]-     - if you built from source, your compiler versions and ideally a build log
[2019-09-17T11:20:04.778+01:00][FATAL]-- If you're working with a numpy git repository, try `git clean -xdf`
[2019-09-17T11:20:04.778+01:00][FATAL]-  (removes all files not under version control) and rebuild numpy.
[2019-09-17T11:20:04.778+01:00][FATAL]-Note: this error has many possible causes, so please don't comment on
[2019-09-17T11:20:04.778+01:00][FATAL]-an existing issue about this - open a new one instead.
[2019-09-17T11:20:04.778+01:00][FATAL]-Original error was: No module named 'numpy.core._multiarray_umath'
[2019-09-17T11:20:04.779+01:00][FATAL]-lambda_runtime.py:371,Failed to initialize Lambda runtime due to exception: Unable to import required dependencies:
[2019-09-17T11:20:04.779+01:00][FATAL]-numpy:
[2019-09-17T11:20:04.779+01:00][FATAL]-IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
[2019-09-17T11:20:04.779+01:00][FATAL]-Importing the numpy c-extensions failed.
[2019-09-17T11:20:04.779+01:00][FATAL]-- Try uninstalling and reinstalling numpy.
[2019-09-17T11:20:04.779+01:00][FATAL]-- If you have already done that, then:
[2019-09-17T11:20:04.779+01:00][FATAL]-  1. Check that you expected to use Python3.7 from "/usr/bin/python3.7",
[2019-09-17T11:20:04.779+01:00][FATAL]-     and that you have no directories in your PATH or PYTHONPATH that can
[2019-09-17T11:20:04.779+01:00][FATAL]-     interfere with the Python and numpy version "1.17.2" you're trying to use.
[2019-09-17T11:20:04.779+01:00][FATAL]-  2. If (1) looks fine, you can open a new issue at
[2019-09-17T11:20:04.779+01:00][FATAL]-     https://github.com/numpy/numpy/issues.  Please include details on:
[2019-09-17T11:20:04.779+01:00][FATAL]-     - how you installed Python
[2019-09-17T11:20:04.779+01:00][FATAL]-     - how you installed numpy
[2019-09-17T11:20:04.779+01:00][FATAL]-     - your operating system
[2019-09-17T11:20:04.779+01:00][FATAL]-     - whether or not you have multiple versions of Python installed
[2019-09-17T11:20:04.779+01:00][FATAL]-     - if you built from source, your compiler versions and ideally a build log
[2019-09-17T11:20:04.779+01:00][FATAL]-- If you're working with a numpy git repository, try `git clean -xdf`
[2019-09-17T11:20:04.779+01:00][FATAL]-  (removes all files not under version control) and rebuild numpy.
[2019-09-17T11:20:04.779+01:00][FATAL]-Note: this error has many possible causes, so please don't comment on
[2019-09-17T11:20:04.779+01:00][FATAL]-an existing issue about this - open a new one instead.
[2019-09-17T11:20:04.779+01:00][FATAL]-Original error was: No module named 'numpy.core._multiarray_umath'
[2019-09-17T11:20:18.818+01:00][FATAL]-lambda_runtime.py:131,Failed to import handler function "doGreenGrassPredictions.function_handler" due to exception: Missing required dependencies ['numpy']
[2019-09-17T11:20:18.819+01:00][FATAL]-lambda_runtime.py:371,Failed to initialize Lambda runtime due to exception: Missing required dependencies ['numpy']
[2019-09-17T11:20:20.21+01:00][FATAL]-lambda_runtime.py:131,Failed to import handler function "doGreenGrassPredictions.function_handler" due to exception: Missing required dependencies ['numpy']
[2019-09-17T11:20:20.211+01:00][FATAL]-lambda_runtime.py:371,Failed to initialize Lambda runtime due to exception: Missing required dependencies ['numpy']

Numpy/Python version information:

Python 3.7
Numpy version unknown since I cannot import numpy to print its version.

It's quite possible this error is related to some proprietary gubbins Amazon is doing, between deploying it with SAM (Serverless Application Model) and then deploying it to an IoT device with IoT GreenGrass, perhaps some weird stuff has happened. I've created this issue so I can link to it in a bug report to Amazon.

[OscarVanL]

I tried following the suggestion from this similar issue, specifically I downloaded the pre-compiled versions of pandas and numpy from here.

This didn't resolve it though, now I get this error:

[2019-09-17T12:26:10.047+01:00][FATAL]-Importing the multiarray numpy extension module failed.  Most
[2019-09-17T12:26:10.047+01:00][FATAL]-likely you are trying to import a failed build of numpy.
[2019-09-17T12:26:10.047+01:00][FATAL]-If you're working with a numpy git repo, try `git clean -xdf` (removes all
[2019-09-17T12:26:10.047+01:00][FATAL]-files not under version control).  Otherwise reinstall numpy.
[2019-09-17T12:26:10.047+01:00][FATAL]-Original error was: No module named 'numpy.core._multiarray_umath'
[2019-09-17T12:26:10.047+01:00][FATAL]-lambda_runtime.py:371,Failed to initialize Lambda runtime due to exception:
[2019-09-17T12:26:10.047+01:00][FATAL]-Importing the multiarray numpy extension module failed.  Most
[2019-09-17T12:26:10.047+01:00][FATAL]-likely you are trying to import a failed build of numpy.
[2019-09-17T12:26:10.047+01:00][FATAL]-If you're working with a numpy git repo, try `git clean -xdf` (removes all
[2019-09-17T12:26:10.047+01:00][FATAL]-files not under version control).  Otherwise reinstall numpy.
[2019-09-17T12:26:10.047+01:00][FATAL]-Original error was: No module named 'numpy.core._multiarray_umath'

[christianoruge]

I have a similar problem when working with PyInstaller to create an app. I get the same error message, but it says "Check that you expected to use Python2.7 from "/usr/bin/python",". this is weird, cause I'm running Python3.7 and I've installed numpy with pip3. When opening "Setup.py" for Numpy I realize it starts with "#!/usr/bin/env python" which should indicate a python 2.7 installation. Changing this line to "#!/usr/bin/env python3.7" solves nothing and the message remains unchanged. My Nympy version is "1.17.2" and I working within a virtual environment.

In console I thing this is the corresponding error:
21:55:14.873294 +0200 PrinterInstallerClientService nw_path_close_fd Failed to close guarded necp fd 6 [9: Bad file descriptor]

When running the app I get:
Traceback (most recent call last):
File "/Users/christianottoruge/Documents/CORals-scripts/CSV-hjelperen.app/Contents/Resources/script", line 11, in
import numpy as np
File "/Users/christianottoruge/Documents/CORals-scripts/CSV-hjelperen.app/Contents/Resources/numpy/init.py", line 142, in
from . import core
File "/Users/christianottoruge/Documents/CORals-scripts/CSV-hjelperen.app/Contents/Resources/numpy/core/init.py", line 47, in
raise ImportError(msg)
ImportError:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy c-extensions failed.

• Try uninstalling and reinstalling numpy.
• If you have already done that, then:
  1. Check that you expected to use Python2.7 from "/usr/bin/python",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.17.2" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues. Please include details on:
     • how you installed Python
     • how you installed numpy
     • your operating system
     • whether or not you have multiple versions of Python installed
     • if you built from source, your compiler versions and ideally a build log
• If you're working with a numpy git repository, try git clean -xdf
  (removes all files not under version control) and rebuild numpy.
  Note: this error has many possible causes, so please don't comment on
  an existing issue about this - open a new one instead.
  Original error was: No module named _multiarray_umath

As you can see I don't pass checkpoint one, but I have no idea about how to change the reference to Python 2.7. I have both 2.7, 3.6 and 3.7 installed and I have a Mac with MacOs Mojave version 10.14.6.

Anyone has a suggestion on how to get around this?

Kind regards, Christian

[iceback]

I'm in the same boat. I'm adding a Lambda Layer with python stats
packages (seaforth,pandas,numpy,scipy). A call to my function needing
those generates pretty much what you have, but no mention of py2.7.

START RequestId: e876b0a0-2cf9-47f6-91df-7b8d798572eb Version: $LATEST
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function':

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.

• Try uninstalling and reinstalling numpy.
• If you have already done that, then:
  1. Check that you expected to use Python3.7 from "/var/lang/bin/python3.7",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.17.2" you're trying to use.

I wonder if it's a shared lib incompatibility? This zip file sent up
as a layer has .so named *.cpython-35m-x86_64-linux-gnu.so

[vishalsharai-archives]

So I was able to solve the problem thanks to the help of the great community here. Using
sudo apt-get install libatlas-base-dev

[mattip]

@OscarVanL were you able to properly install numpy into your lambda instance?

[iceback]

Yes! It all came down to installing the packages on a matching instance type using the --target=<dir> argument for pip3. Then zip that dir and install/upload via lambda console The path in the zip file must start with “python/“.

…

On Dec 11, 2019, at 9:44 AM, Matti Picus ***@***.***> wrote: @OscarVanL <https://github.com/OscarVanL> were you able to properly install numpy into your lambda instance? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#14532?email_source=notifications&email_token=AACY7PB6AKKH5AQD6WUBJCLQYEKH3A5CNFSM4IXN3XRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGTZOGA#issuecomment-564631320>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACY7PG2PTGMHFY7SQRUKLDQYEKH3ANCNFSM4IXN3XRA>.

[mattip]

@iceback thanks for the update and hint, but this issue was opened by @OscarVanL

[OscarVanL]

Apologies for never replying to this issue. I'm no longer working on this, so can't say if the previously mentioned workarounds work for me or not.
But judging by the multiple responses saying it's working, I'm going to close the issue.

[AnandShiva]

@vishalsharai-trail how did you install sudo apt-get install libatlas-base-dev in lambda function?

[mateusfbsoares]

I had a similar problem and fixed it this way

[Shaunak-Pandit]

Use Klayers package from this link- https://github.com/keithrozario/Klayers

[shawnesquivel]

I had a similar problem and fixed it this way

my form looks different from yours, how did you fix it?