sudo yum -y update
sudo yum-config-manager --enable epel
sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel
cd /tmp
curl -L http://download.osgeo.org/gdal/2.0.0/gdal-2.0.0.tar.gz | tar zxf -
cd gdal-2.0.0/
./configure --prefix=/usr/local --without-python
make -j4
sudo make install
cd /usr/local
tar zcvf ~/gdal-2.0.0-amz1.tar.gz *
Last active
May 17, 2024 12:53
-
-
Save mojodna/2f596ca2fca48f08438e to your computer and use it in GitHub Desktop.
GDAL 2.0 on Amazon Linux
We are using this to run GDAL on Amazon Linux.
Since compiling takes quite a long time, you probably won't want to recompile each time an EC2 loads up (we're using Elastic Beanstalk's autoscaling).
If you host the tar file you create as well aslibproj.so(in/usr/lib64) you can add container commands to avoid recompiling each time.
Rough idea of what is working so far:04_install_gdal: command: | sudo wget [url]/gdal_extract.tgz 05_install_gdal2: command: | sudo wget [url]libproj.so 06_install_gdal3: command: | sudo tar -xvf gdal_extract.tgz -C /usr/local/ 07_install_gdal4: command: | sudo cp libproj.so /usr/lib64/Can you share the full code for this with the commands to avoid recompiling each time?
This is essentially the full code (these are container commands for Elastic Beanstalk). I have removed the URL's because they are hosted on our public S3. You will need to follow the instructions in the original post to produce a tar.gz, then upload that tgz and paste the URL for the 04_install_gdal command. Then, you'll need to copy /usr/lib64/libproj.so and host that on S3 also, and paste that URL for 05_install_gdal2. Hope that helps.
Do the whole process on an EC2 instance that is identical to the ones that will be autoscaled.
Thanks a bunch for this post, very helpful. I've used this successfully on Elastic Beanstalk. The only thing I'd add is that EB now has a "test" parameter for commands that can potentially save needless downloads (or maybe the parameter option has been around for awhile, I just discovered it):
04_install_gdal:
command: sudo wget [url]/gdal_extract.tgz
test: "[ ! -f gdal_extract.tgz ]"
05_install_gdal2:
command: sudo wget [url]libproj.so
test: "[ ! -f /usr/lib64/libproj.so ]"
06_install_gdal3:
command: sudo tar -k -xvf gdal_extract.tgz -C /usr/local
test: "[ ! -f /usr/local/lib/libgdal.so ]"
07_install_gdal4:
command: sudo cp libproj.so /usr/lib64/
test: "[ ! -f /usr/lib64/libproj.so ]"
So the commands will only run if the test evaluates to true (or returns 0). For 06_install_gdal3 I just check for the existence of libgdal.so, which presumably won't be there without a GDAL installation. Also I use the -k flag on the tar command to avoid replacing existing files and blowing up the existing /usr/local directory.
Anyone willing to provide some support my way? I'm hitting an error deploying new ec2 instances, and any help would be greatly appreciated!
My Goal:
Precompiled gdal to run in elastic beanstalk so that spinning up a new server does not take 10-15 minutes (same goal as mentioned above)
I am currently trying the pre deploy method used by @nashmaniac and will likely hit a timeout error - the build time for gdal is quite long on smaller ec2 instances, hence the above goal.
I've created the 2 tar.gz files (per original post et al):
sudo yum -y update
sudo yum-config-manager --enable epel
sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel s3cmd gdal-bin
cd /tmp
curl -L http://download.osgeo.org/gdal/2.4.4/gdal-2.4.4.tar.gz | tar zxf -
cd gdal-2.4.4/
./configure --prefix=/usr/local --without-python
make -j4
sudo make install
sudo ldconfig
cd /usr/local
tar zcvf ~/gdal-bundle.tar.gz *
cd /usr/lib64/
tar zcvf ~/libproj-bundle.tar.gz ./libproj.soThese are moved to S3 using s3cmd
Following the advice from @dliff and @essy2017, in Elastic Beanstalk .ebextensions I've got the container commands fetching and unpacking:
container_commands:
03_set_cwd:
command: cd /tmp/
04_gdal_copy_s3:
command: wget [s3-url]/gdal-full-bundle.tar.gz
05_libproj_copy_s3:
command: wget [s3-url]/libproj-bundle.tar.gz
06_gdal_unpack:
command: tar -xvf gdal-full-bundle.tar.gz -C /usr/local/
07_libproj_unpack:
command: tar -xvf libproj-bundle.tar.gz -C /usr/lib64/
10_collectstatic:
command: "source /opt/python/run/venv/bin/activate && python manage.py collectstatic --noinput"
12_migrate:
command: "source /opt/python/run/venv/bin/activate && python manage.py migrate --noinput"
leader_only: trueBut eb deploy keeps failing at 12_migrate throwing the error
...
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/db/models/fields.py", line 3, in <module>
from django.contrib.gis import forms, gdal
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module>
from .fields import ( # NOQA
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/forms/fields.py", line 2, in <module>
from django.contrib.gis.gdal import GDALException
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
from django.contrib.gis.gdal.datasource import DataSource
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/gdal/datasource.py", line 39, in <module>
from django.contrib.gis.gdal.driver import Driver
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/gdal/libgdal.py", line 46, in <module>
lgdal = CDLL(lib_path)
File "/usr/lib64/python3.6/ctypes/__init__.py", line 343, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libgdal.so.20: cannot open shared object file: No such file or directory
SSH into EC2 and run gdalinfo --version throws this:
gdalinfo: error while loading shared libraries: libgeos_c.so.1: cannot open shared object file: No such file or directory
Notes:
- I had
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATHas a container command to no avail. I've also confirmed the var's value in the shell to be /user/local/lib - adding
sudoto each container command did not work - I tried ssh'ing in and linking libgdal.so.20 using
ln -s /usr/lib/libgdal.so /usr/lib/libgdal.so.1but it didn't help, same error
I seem to be going in circles for answers, and I'm hoping someone encountered what I'm seeing!
Also, is it just bad practice to copy over the whole /usr/local folder?
@seanwhudson
I actually encountered errors myself after deployment. I'm not sure about your current instance, but in your last note creating the simlink for libgdal.so.20 did you have a typo? The OSError looks like you want to link to libgdal.so.20 not libgdal.so.1.
I have found a new solution after starting from scratch by modifying install and compile scripts I found at https://gist.github.com/hervenivon/fe3a327bc28b142e51beb38ef11844c0.
[1] Create a script install-gdal.sh:
#!/bin/bash
# Installation of GDAL and dependencies.
# Based on https://gist.github.com/hervenivon/fe3a327bc28b142e51beb38ef11844c0
# Update with your versions.
GEOS_VERSION=3.7.1
GDAL_VERSION=2.3.3
PROJ4_VERSION=4.9.3
sudo yum-config-manager --enable epel
sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel
# Compilation work for geos.
mkdir -p "/tmp/geos-${GEOS_VERSION}-build"
cd "/tmp/geos-${GEOS_VERSION}-build"
curl -o "geos-${GEOS_VERSION}.tar.bz2" \
"http://download.osgeo.org/geos/geos-${GEOS_VERSION}.tar.bz2" \
&& bunzip2 "geos-${GEOS_VERSION}.tar.bz2" \
&& tar xvf "geos-${GEOS_VERSION}.tar"
cd "/tmp/geos-${GEOS_VERSION}-build/geos-${GEOS_VERSION}"
./configure --prefix=/usr/local/geos
# Make in parallel with 2x the number of processors.
make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \
&& sudo make install \
&& sudo ldconfig
# Compilation work for proj4.
mkdir -p "/tmp/proj-${PROJ4_VERSION}-build"
cd "/tmp/proj-${PROJ4_VERSION}-build"
curl -o "proj-${PROJ4_VERSION}.tar.gz" \
"http://download.osgeo.org/proj/proj-${PROJ4_VERSION}.tar.gz" \
&& tar xfz "proj-${PROJ4_VERSION}.tar.gz"
cd "/tmp/proj-${PROJ4_VERSION}-build/proj-${PROJ4_VERSION}"
./configure --prefix=/usr/local/proj4
# Make in parallel with 2x the number of processors.
make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \
&& sudo make install \
&& sudo ldconfig
# Compilation work for GDAL
mkdir -p "/tmp/gdal-${GDAL_VERSION}-build"
cd "/tmp/gdal-${GDAL_VERSION}-build"
curl -o "gdal-${GDAL_VERSION}.tar.gz" \
"http://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION}.tar.gz" \
&& tar xfz "gdal-${GDAL_VERSION}.tar.gz"
cd "/tmp/gdal-${GDAL_VERSION}-build/gdal-${GDAL_VERSION}"
./configure --prefix=/usr/local/gdal \
--with-static-proj4=/usr/local/proj4 \
--without-python
# Make in parallel with 2x the number of processors.
make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \
&& sudo make install \
&& sudo ldconfig
# Bundle resources.
cd /usr/local/geos
tar zcvf "/tmp/geos-${GEOS_VERSION}.tar.gz" *
cd /usr/local/proj4
tar zcvf "/tmp/proj4-${PROJ4_VERSION}.tar.gz" *
cd /usr/local/gdal
tar zcvf "/tmp/gdal-${GDAL_VERSION}.tar.gz" *
[2] Run the script on an EC2 instance (takes a long time) after which you should have the three bundled files available:
/tmp/geos-3.7.1.tar.gz
/tmp/proj4-4.9.3.tar.gz
/tmp/gdal-2.3.3.tar.gz
[3] Upload these to your S3 bucket.
[4] Add the following to your .ebextensions. Each command tests for the existence of the directory on your instance and downloads and unzips the source as needed.
commands:
01_install_geos:
command: |
sudo wget [s3-url]/geos-3.7.1.tar.gz
sudo mkdir -p /usr/local/geos
sudo tar -xvf geos-3.7.1.tar.gz -C /usr/local/geos
sudo rm -f geos-3.7.1.tar.gz
test: "[ ! -d /usr/local/geos ]"
02_install_proj4:
command: |
sudo wget [s3-url]/proj4-4.9.3.tar.gz
sudo mkdir -p /usr/local/proj4
sudo tar -xvf proj4-4.9.3.tar.gz -C /usr/local/proj4
sudo rm -f proj4-4.9.3.tar.gz
test: "[ ! -d /usr/local/proj4 ]"
03_install_gdal:
command: |
sudo wget [s3-url]/gdal-2.3.3.tar.gz
sudo mkdir -p /usr/local/gdal
sudo tar -xvf gdal-2.3.3.tar.gz -C /usr/local/gdal
sudo rm -f gdal-2.3.3.tar.gz
test: "[ ! -d /usr/local/gdal ]"
[5] Finally, add the following to your .ebextensions/options.config file:
option_settings:
aws:elasticbeanstalk:application:environment:
PATH: /usr/local/gdal/bin:$PATH
LD_LIBRARY_PATH: /usr/local/proj4/lib:/usr/local/gdal/lib:$LD_LIBRARY_PATH
GDAL_DATA: /usr/local/gdal/share/gdal
Seems like a lot of steps now that I've typed it all out but it has been working reliably for me on EB running Python 3.6 running on 64bit Amazon Linux/2.9.4. Seems much cleaner than bundling the entire /usr/local directory too.
This is excellent, thank you @essy2017. I'll take a run and post any issues I work through as well.
I'm very green to the Linux (and aws's Centos) env, so I was unsure of how to bundle just geos/gdal/proj and transport them to a different system - and even if it was possible! From the script it looks like just taking the /usr/local/{lib} folders is sufficient, so that's good to know.
Okay, managed to get it working! Just two small fixes to your option_settings
option_settings:
aws:elasticbeanstalk:application:environment:
PATH: /usr/local/gdal/bin:$PATH
LD_LIBRARY_PATH: /usr/local/proj4/lib:/usr/local/geos/lib:/usr/local/gdal/lib:$LD_LIBRARY_PATH
GDAL_LIBRARY_PATH: /usr/local/gdal/lib/libgdal.so
Note the addition of /usr/local/geos/lib: in LD_LIBRARY_PATH and changing GDAL_DATA to GDAL_LIBRARY_PATH - while the second may have worked, the first change to add the geos libs was what finally turned the corner for me.
One thing I would caution is that eb runs .ebextension files in alphabetical order, so it might be worth it to either merge the above two files, or prefix their names with 01 and 02 to avoid any strange behavior.
Lastly, bcs I never answered before re: typo - No; it was throwing an error looking for [...].1 shared object files, which helped me pin down the missing geos path.
Learned a ton here, thanks again for the assist.
Thank you @essy2017 and @vancityhuddy. You saved my day.
For anyone else I've created a small example repo. Hope it helps!
I set up the script and .ebextension/01_options.conf file from the previous posts, but now I am getting the error:
from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/gdal/libgdal.py", line 47, in <module>
lgdal = CDLL(lib_path)
File "/usr/lib64/python3.6/ctypes/__init__.py", line 343, in __init__
self._handle = _dlopen(self._name, mode)
`OSError: libcrypto.so.1.1: cannot open shared object file: No such file or directory`
From my understanding, the path to that file is invalid. Here is my 01_options.conf:
...
option_settings:
aws:elasticbeanstalk:application:environment:
PATH: /usr/local/gdal/bin:$PATH
LD_LIBRARY_PATH: /usr/local/proj4/lib:/usr/local/geos/lib:/usr/local/gdal/lib:$LD_LIBRARY_PATH
GDAL_LIBRARY_PATH: /usr/local/gdal/lib/libgdal.so
Vancityhuddy and essy2017, this is very much appreciated.
I ran into something similar, and iirc it had to do with the gdal versions and/or the output names I was using. Don't go for latest osgeo libraries, I think that caused an issue for me. You'll see in my gdal-build-script from the linked repo that I set the versions to
GEOS_VERSION=3.6.4
GDAL_VERSION=2.4.4
PROJ4_VERSION=5.2.0
A quicker solution (bcs you don't have to recompile) is to verify your lib names. SSH into your ec2 instance, rummage around your /usr/local/ folders or search for which lib filenames exist on the machine. Then you could update your 01_options.conf to match the filenames.
I recompiled with those versions, but no luck.
I tried searching through the EC2 using ssh, but I only found the following:
Location: /usr/lib64/
libcrypto.so
libcrypto.so.10
libcrypto.so.1.0.2k
libcrypt.so
Location: /lib64/
libcrypt.so.1
I don't think Libcrypto.so.1.1 is installed on this EC.
My AWS EBS application seems to failing here when trying to execute this method in GeoDjango:
https://github.com/django/django/blob/86908785076b2bbc31b908781da6b6ad1779b18b/django/contrib/gis/gdal/libgdal.py#L46
Vancity, does your Amazon Linux have libcrypto.so.1.1?
this is very very very slow...
this is very very very slow...
@hanayashiki
It is very dependent on what type of machine you put into play. I'd recommend not going less than a T3.small - if you run a micro or nano it will take forever!
@seanwhudson
I actually encountered errors myself after deployment. I'm not sure about your current instance, but in your last note creating the simlink for
libgdal.so.20did you have a typo? The OSError looks like you want to link tolibgdal.so.20notlibgdal.so.1.I have found a new solution after starting from scratch by modifying install and compile scripts I found at https://gist.github.com/hervenivon/fe3a327bc28b142e51beb38ef11844c0.
[1] Create a script
install-gdal.sh:#!/bin/bash # Installation of GDAL and dependencies. # Based on https://gist.github.com/hervenivon/fe3a327bc28b142e51beb38ef11844c0 # Update with your versions. GEOS_VERSION=3.7.1 GDAL_VERSION=2.3.3 PROJ4_VERSION=4.9.3 sudo yum-config-manager --enable epel sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel # Compilation work for geos. mkdir -p "/tmp/geos-${GEOS_VERSION}-build" cd "/tmp/geos-${GEOS_VERSION}-build" curl -o "geos-${GEOS_VERSION}.tar.bz2" \ "http://download.osgeo.org/geos/geos-${GEOS_VERSION}.tar.bz2" \ && bunzip2 "geos-${GEOS_VERSION}.tar.bz2" \ && tar xvf "geos-${GEOS_VERSION}.tar" cd "/tmp/geos-${GEOS_VERSION}-build/geos-${GEOS_VERSION}" ./configure --prefix=/usr/local/geos # Make in parallel with 2x the number of processors. make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \ && sudo make install \ && sudo ldconfig # Compilation work for proj4. mkdir -p "/tmp/proj-${PROJ4_VERSION}-build" cd "/tmp/proj-${PROJ4_VERSION}-build" curl -o "proj-${PROJ4_VERSION}.tar.gz" \ "http://download.osgeo.org/proj/proj-${PROJ4_VERSION}.tar.gz" \ && tar xfz "proj-${PROJ4_VERSION}.tar.gz" cd "/tmp/proj-${PROJ4_VERSION}-build/proj-${PROJ4_VERSION}" ./configure --prefix=/usr/local/proj4 # Make in parallel with 2x the number of processors. make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \ && sudo make install \ && sudo ldconfig # Compilation work for GDAL mkdir -p "/tmp/gdal-${GDAL_VERSION}-build" cd "/tmp/gdal-${GDAL_VERSION}-build" curl -o "gdal-${GDAL_VERSION}.tar.gz" \ "http://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION}.tar.gz" \ && tar xfz "gdal-${GDAL_VERSION}.tar.gz" cd "/tmp/gdal-${GDAL_VERSION}-build/gdal-${GDAL_VERSION}" ./configure --prefix=/usr/local/gdal \ --with-static-proj4=/usr/local/proj4 \ --without-python # Make in parallel with 2x the number of processors. make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \ && sudo make install \ && sudo ldconfig # Bundle resources. cd /usr/local/geos tar zcvf "/tmp/geos-${GEOS_VERSION}.tar.gz" * cd /usr/local/proj4 tar zcvf "/tmp/proj4-${PROJ4_VERSION}.tar.gz" * cd /usr/local/gdal tar zcvf "/tmp/gdal-${GDAL_VERSION}.tar.gz" *[2] Run the script on an EC2 instance (takes a long time) after which you should have the three bundled files available:
/tmp/geos-3.7.1.tar.gz /tmp/proj4-4.9.3.tar.gz /tmp/gdal-2.3.3.tar.gz[3] Upload these to your S3 bucket.
[4] Add the following to your
.ebextensions. Each command tests for the existence of the directory on your instance and downloads and unzips the source as needed.commands: 01_install_geos: command: | sudo wget [s3-url]/geos-3.7.1.tar.gz sudo mkdir -p /usr/local/geos sudo tar -xvf geos-3.7.1.tar.gz -C /usr/local/geos sudo rm -f geos-3.7.1.tar.gz test: "[ ! -d /usr/local/geos ]" 02_install_proj4: command: | sudo wget [s3-url]/proj4-4.9.3.tar.gz sudo mkdir -p /usr/local/proj4 sudo tar -xvf proj4-4.9.3.tar.gz -C /usr/local/proj4 sudo rm -f proj4-4.9.3.tar.gz test: "[ ! -d /usr/local/proj4 ]" 03_install_gdal: command: | sudo wget [s3-url]/gdal-2.3.3.tar.gz sudo mkdir -p /usr/local/gdal sudo tar -xvf gdal-2.3.3.tar.gz -C /usr/local/gdal sudo rm -f gdal-2.3.3.tar.gz test: "[ ! -d /usr/local/gdal ]"[5] Finally, add the following to your
.ebextensions/options.configfile:option_settings: aws:elasticbeanstalk:application:environment: PATH: /usr/local/gdal/bin:$PATH LD_LIBRARY_PATH: /usr/local/proj4/lib:/usr/local/gdal/lib:$LD_LIBRARY_PATH GDAL_DATA: /usr/local/gdal/share/gdalSeems like a lot of steps now that I've typed it all out but it has been working reliably for me on EB running
Python 3.6 running on 64bit Amazon Linux/2.9.4. Seems much cleaner than bundling the entire/usr/localdirectory too.
Hey,
I am using your solution with Amazon Linux 2 and Python 3.7 but facing issue in deployment. Can you please help for Amazon Linux 2
@rahulkhairnarr What is the issue/can you share error messages? Also, facing issues on EB deployment?
this is very very very slow...
@hanayashiki
It is very dependent on what type of machine you put into play. I'd recommend not going less than a T3.small - if you run a micro or nano it will take forever!
Not forever, just a night's sleep
It worked for me with gdal v2.2.0, django 2.1
I had to change the GDAL path: GDAL_LIBRARY_PATH = /usr/local/lib/libgdal.so in settings.py
Attempting install on a sagemaker notebook instance, make -j4 completes fine but the following sudo make installresults in a massive volume of errors like:
/opt/conda/include/unicode/localpointer.h:224:5: warning: identifier ‘noexcept’ is a keyword in C++11 [-Wc++0x-compat]
LocalPointer(LocalPointer<T> &&src) U_NOEXCEPT : LocalPointerBase<T>(src.ptr) {
Hello, thank you very much for the information I wanted to ask a favor if it is possible to share these 3 files to do a test with sagemaker please
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Can you share the full code for this with the commands to avoid recompiling each time?