-
-
Save niranjv/f80fc1f488afc49845e2ff3d5df7f83b to your computer and use it in GitHub Desktop.
| # A virtualenv running Python3.6 on Amazon Linux/EC2 (approximately) simulates the Python 3.6 Docker container used by Lambda | |
| # and can be used for developing/testing Python 3.6 Lambda functions | |
| # This script installs Python 3.6 on an EC2 instance running Amazon Linux and creates a virtualenv running this version of Python | |
| # This is required because Amazon Linux does not come with Python 3.6 pre-installed | |
| # and several packages available in Amazon Linux are not available in the Lambda Python 3.6 runtime | |
| # The script has been tested successfully on a t2.micro EC2 instance (Root device type: ebs; Virtualization type: hvm) | |
| # running Amazon Linux AMI 2017.03.0 (HVM), SSD Volume Type - ami-c58c1dd3 | |
| # and was developed with the help of AWS Support | |
| # The steps in this script are: | |
| # - install pre-reqs | |
| # - install Python 3.6 | |
| # - create virtualenv | |
| # install pre-requisites | |
| sudo yum -y groupinstall development | |
| sudo yum -y install zlib-devel | |
| sudo yum -y install openssl-devel | |
| # Installing openssl-devel alone seems to result in SSL errors in pip (see https://medium.com/@moreless/pip-complains-there-is-no-ssl-support-in-python-edbdce548852) | |
| # Need to install OpenSSL also to avoid these errors | |
| wget https://github.com/openssl/openssl/archive/OpenSSL_1_0_2l.tar.gz | |
| tar -zxvf OpenSSL_1_0_2l.tar.gz | |
| cd openssl-OpenSSL_1_0_2l/ | |
| ./config shared | |
| make | |
| sudo make install | |
| export LD_LIBRARY_PATH=/usr/local/ssl/lib/ | |
| cd .. | |
| rm OpenSSL_1_0_2l.tar.gz | |
| rm -rf openssl-OpenSSL_1_0_2l/ | |
| # Install Python 3.6 | |
| wget https://www.python.org/ftp/python/3.6.0/Python-3.6.0.tar.xz | |
| tar xJf Python-3.6.0.tar.xz | |
| cd Python-3.6.0 | |
| ./configure | |
| make | |
| sudo make install | |
| cd .. | |
| rm Python-3.6.0.tar.xz | |
| sudo rm -rf Python-3.6.0 | |
| # Create virtualenv running Python 3.6 | |
| sudo pip install --upgrade virtualenv | |
| virtualenv -p python3 MYVENV | |
| source MYVENV/bin/activate | |
| python --version | |
| # Python 3.6.0 |
Would it work to turn this script into a Dockerfile and do testing as a container, rather than spinning up an EC2 instance?
There is an amazonlinux container available with tags for 2018xxxx (Amazon Linux 1) and the default/latest points to Amazon Linux 2 I believe.
You can enable epel for the amazonlinux container using the amazon-linux-extras install epel and then when you yum install things or yum search it should be able to access most things from epel as well.
If you are doing this in Docker for a deployable "builder" you probably don't want to use yum install @development as that pulls in a huge amount of packages (123 when I just ran it, weighing in at 121mb downloaded and ~141mb installed according to an uninstall, while git seems to pull in about half of these, there are quite a few other likely unused packages as well). Using @development or yum groupinstall development might be less tedious if you are in hurry than honing in on only the packages you need to compile Python and/or native extensions by fixing failures one at a time.
Personally I'm both lazy and OCD so I've gotten tired of handling the various quirks as new Python versions come out and just use Pyenv to install different versions of Python than what are available in the official repositories (on many distros and macOS and Windows).
Sadly it appears in our specific use case we ended up needing to use centos:7 for our base image because the version of gcc coming from amazonlinux was having issues compiling our code where the centos:7 version worked consistently.
FROM centos:7
RUN yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
RUN yum install -y \
# shared build deps
make \
gcc-c++ \
git \
# Python C extension deps
proj-devel \
unzip \
autoconf \
automake \
libtool \
cmake \
proj-epsg \
libpng \
libpng-devel \
# pyenv/python only build deps
xz-devel \
zlib-devel \
bzip2-devel \
readline-devel \
sqlite sqlite-devel \
openssl-devel \
libffi-devel \
yum-utils \
&& yum clean all
# Install specific python 3 version and use that pip3 to get some pre-reqs
# Maybe don't add --enable-optimizations unless you are actually running intensive
# python apps in the container as it takes a looooong time to compile
# If only using pyenv for this initial Python install, no need to put the PATH and
# pyenv init things into the .bashrc for the root user or a container specific user
RUN curl https://pyenv.run | bash \
&& export PATH="/root/.pyenv/bin:$PATH" \
&& env PYTHON_CONFIGURE_OPTS="--enable-shared" $(pyenv which python-build) 3.6.10 /usr/local/ \
&& pip3 install awscli
Skeleton of an AmazonLinux Dockerfile, can basically swap out the first 3 lines of the centos version with this and get a similar result.
FROM amazonlinux
# The amazon-linux-extras enables epel instead of installing an rpm from a URL
RUN amazon-linux-extras install epel && \
yum install -y \
# Convenience group that adds a bit too much
# @development \
git \
yum-utils \
&& yum clean all
Here's an easy copy-paste install script for Python 3.6.12 on the default Amazon Linux 2 EC2 instance, based on the replies above.
sudo yum -y groupinstall development
sudo yum -y install zlib-devel
sudo yum -y install openssl-devel bzip2-devel
sudo yum remove python
curl -O https://www.python.org/ftp/python/3.6.12/Python-3.6.12.tgz
tar xzvf Python-3.6.12.tgz
cd Python-3.6.12
sudo ./configure --enable-optimizations --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
sudo make install
Then run Python3 with:
python3
And use pip with:
python3 -m pip
Thank you so much !
In addition, on step #8, the
--enable-sharedflag can/should be included as well. Leaving it out can yield the following error message if a package depends on packages that are cpython-based (I think that's why, anyway. In my case I'm working with a C++ library that has a Python wrapper).ImportError: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory