How to make a Python pandas Layer for AWS Lambda

aws_pandas_layer.md

How to make pandas as an AWS layer

1. Install pandas to local directory with pip: pip install -t . pandas

2. Remove pandas and numpy directories rm -r pandas numpy

3. Download the Linux distribution for pandas (choose the Python version that you want to use): https://pypi.org/project/pandas/#files

4. Download the Linux distribution for numpy (must be the same as the pandas one): https://pypi.org/project/numpy/#files

5. Move the downloaded files to your current directory and unzip them: unzip numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl and unzip pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl

   a. Note: Lambda with Python 3.7 requires pytz lib: $ pip install -t . pytz

6. Remove unecessary files: rm -r *.whl *.dist-info __pycache__

7. Make the following directory (change the version of Python according to what you downloaded earlier): lambda_layers/python/lib/python3.8/site-packages/.

8. Move your whole directory into the site-packages folder.

9. Go to cd lambda_layers and run: zip -r pandas_lambda_layer.zip *

10. Upload the zip to S3.

11. Create a Layer that uses the file uploaded to S3.

12. Create a Lambda function (must uses the same Python version than before).

13. You can test that everything works by using the code in lambda_function.py

14. Have fun with pandas!

Sources:

• https://medium.com/@korniichuk/lambda-with-pandas-fd81aa2ff25e

• https://towardsdatascience.com/introduction-to-amazon-lambda-layers-and-boto3-using-python3-39bd390add17

After a few tried, I got this error

Response
{
  "errorMessage": "Unable to import module 'lambda_function': Error importing numpy: you should not try to import numpy from\n        its source directory; please exit the numpy source tree, and relaunch\n        your python interpreter from there.",
  "errorType": "Runtime.ImportModuleError",
  "stackTrace": []
}

Is it expected?

@amzar96 Did you managed to fic the issue? I am facing the similar.

@amzar96 Did you managed to fic the issue? I am facing the similar.

You must use the same Python version you define when creating the layer.

1. Create virtualenv in local
2. pip install libraries
3. follow the above step