GDAL based AWS Lambda Layers

AWS lambda (Amazonlinux) Layers and Docker images.

Python package

numpy

pygeos==0.8
shapely==1.7.1
rasterio==1.1.8

pyproj==2.4.1  # Only for GDAL>=3.1

mercantile==1.1.6
supermercado==0.2.0

# HACK, those package are in the docker image but not in the lambda env
requests
pyyaml

Available layers

gdal	size (Mb)	unzipped size (Mb)	arn
3.2	43.8	138.8	arn:aws:lambda:{REGION}:524387336408:layer:gdal32-python37-geo:{VERSION}
3.1	43.7	128.4	arn:aws:lambda:{REGION}:524387336408:layer:gdal31-python37-geo:{VERSION}
2.4	36.3	121.3	arn:aws:lambda:{REGION}:524387336408:layer:gdal24-python37-geo:{VERSION}
---	---	---	---
3.2	44.4	140	arn:aws:lambda:{REGION}:524387336408:layer:gdal32-python38-geo:{VERSION}
3.1	44.3	139.7	arn:aws:lambda:{REGION}:524387336408:layer:gdal31-python38-geo:{VERSION}
2.4	36.7	130	arn:aws:lambda:{REGION}:524387336408:layer:gdal24-python38-geo:{VERSION}

See full list of ARN

Arns format

• arn:aws:lambda:${region}:524387336408:layer:gdal(24|31|32)-python(37|38)-geo:${version}

Regions

• ap-northeast-1
• ap-northeast-2
• ap-south-1
• ap-southeast-1
• ap-southeast-2
• ca-central-1
• eu-central-1
• eu-north-1
• eu-west-1
• eu-west-2
• eu-west-3
• sa-east-1
• us-east-1
• us-east-2
• us-west-1
• us-west-2

Layer content

layer.zip
  |
  |___ bin/      # Binaries
  |___ lib/      # Shared libraries (GDAL, PROJ, GEOS...)
  |___ share/    # GDAL/PROJ data directories
  |___ python/   # Python modules

The layer content will be unzip in /opt directory in AWS Lambda. For the python libs to be able to use the C libraries you have to make sure to set 2 important environment variables:

• GDAL_DATA: /opt/share/gdal
• PROJ_LIB: /opt/share/proj

How To

There are 2 ways to use the layers:

1. Simple (No dependencies)

If you don't need to add more python module (dependencies), you can just create a lambda package (zip file) with you lambda handler.

zip -r9q package.zip handler.py

Content:

package.zip
  |___ handler.py   # aws lambda python handler

AWS Lambda Config:

• arn: arn:aws:lambda:us-east-1:524387336408:layer:gdal32-python38-geo:1 (example)
• env:
  • GDAL_DATA: /opt/share/gdal
  • PROJ_LIB: /opt/share/proj
• lambda handler: handler.handler

2. Advanced (need other python dependencies)

If your lambda handler needs more dependencies you'll have to use the exact same environment. To ease this you can find the docker images for each lambda on docker hub.

• Create a docker file

FROM lambgeo/lambda-gdal:3.2-python3.8-geo

ENV PYTHONUSERBASE=/var/task

# Install dependencies
COPY handler.py $PYTHONUSERBASE/handler.py

# Here we use the `--user` option to make sure to not replicate modules.
RUN pip install rio-tiler --user

# Move some files around
RUN mv ${PYTHONUSERBASE}/lib/python3.8/site-packages/* ${PYTHONUSERBASE}/
RUN rm -rf ${PYTHONUSERBASE}/lib

echo "Create archive"
RUN cd $PYTHONUSERBASE && zip -r9q /tmp/package.zip *

• create package

docker build --tag package:latest .
docker run --name lambda -w /var/task -itd package:latest bash
docker cp lambda:/tmp/package.zip package.zip
docker stop lambda
docker rm lambda

Content:

package.zip
  |___ handler.py   # aws lambda python handler
  |___ module1/     # dependencies
  |___ module2/
  |___ module3/
  |___ ...

AWS Lambda Config:

• arn: arn:aws:lambda:us-east-1:524387336408:layer:gdal32-python38-geo:1 (example)
• env:
  • GDAL_DATA: /opt/share/gdal
  • PROJ_LIB: /opt/share/proj
• lambda handler: handler.handler

Refactor

We recently refactored the repo, to see old documentation please refer to https://github.com/lambgeo/geo-layer/tree/60c9cf69a4529e14d4394a0a3e78dd5f84d9e6ec