sudo yum -y update
sudo yum-config-manager --enable epel
sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel
cd /tmp
curl -L http://download.osgeo.org/gdal/2.0.0/gdal-2.0.0.tar.gz | tar zxf -
cd gdal-2.0.0/
./configure --prefix=/usr/local --without-python
make -j4
sudo make install
cd /usr/local
tar zcvf ~/gdal-2.0.0-amz1.tar.gz *
Last active
May 17, 2024 12:53
-
-
Save mojodna/2f596ca2fca48f08438e to your computer and use it in GitHub Desktop.
GDAL 2.0 on Amazon Linux
This is excellent, thank you @essy2017. I'll take a run and post any issues I work through as well.
I'm very green to the Linux (and aws's Centos) env, so I was unsure of how to bundle just geos/gdal/proj and transport them to a different system - and even if it was possible! From the script it looks like just taking the /usr/local/{lib} folders is sufficient, so that's good to know.
Okay, managed to get it working! Just two small fixes to your option_settings
option_settings:
aws:elasticbeanstalk:application:environment:
PATH: /usr/local/gdal/bin:$PATH
LD_LIBRARY_PATH: /usr/local/proj4/lib:/usr/local/geos/lib:/usr/local/gdal/lib:$LD_LIBRARY_PATH
GDAL_LIBRARY_PATH: /usr/local/gdal/lib/libgdal.so
Note the addition of /usr/local/geos/lib: in LD_LIBRARY_PATH and changing GDAL_DATA to GDAL_LIBRARY_PATH - while the second may have worked, the first change to add the geos libs was what finally turned the corner for me.
One thing I would caution is that eb runs .ebextension files in alphabetical order, so it might be worth it to either merge the above two files, or prefix their names with 01 and 02 to avoid any strange behavior.
Lastly, bcs I never answered before re: typo - No; it was throwing an error looking for [...].1 shared object files, which helped me pin down the missing geos path.
Learned a ton here, thanks again for the assist.
Thank you @essy2017 and @vancityhuddy. You saved my day.
For anyone else I've created a small example repo. Hope it helps!
I set up the script and .ebextension/01_options.conf file from the previous posts, but now I am getting the error:
from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/gdal/libgdal.py", line 47, in <module>
lgdal = CDLL(lib_path)
File "/usr/lib64/python3.6/ctypes/__init__.py", line 343, in __init__
self._handle = _dlopen(self._name, mode)
`OSError: libcrypto.so.1.1: cannot open shared object file: No such file or directory`
From my understanding, the path to that file is invalid. Here is my 01_options.conf:
...
option_settings:
aws:elasticbeanstalk:application:environment:
PATH: /usr/local/gdal/bin:$PATH
LD_LIBRARY_PATH: /usr/local/proj4/lib:/usr/local/geos/lib:/usr/local/gdal/lib:$LD_LIBRARY_PATH
GDAL_LIBRARY_PATH: /usr/local/gdal/lib/libgdal.so
Vancityhuddy and essy2017, this is very much appreciated.
I ran into something similar, and iirc it had to do with the gdal versions and/or the output names I was using. Don't go for latest osgeo libraries, I think that caused an issue for me. You'll see in my gdal-build-script from the linked repo that I set the versions to
GEOS_VERSION=3.6.4
GDAL_VERSION=2.4.4
PROJ4_VERSION=5.2.0
A quicker solution (bcs you don't have to recompile) is to verify your lib names. SSH into your ec2 instance, rummage around your /usr/local/ folders or search for which lib filenames exist on the machine. Then you could update your 01_options.conf to match the filenames.
I recompiled with those versions, but no luck.
I tried searching through the EC2 using ssh, but I only found the following:
Location: /usr/lib64/
libcrypto.so
libcrypto.so.10
libcrypto.so.1.0.2k
libcrypt.so
Location: /lib64/
libcrypt.so.1
I don't think Libcrypto.so.1.1 is installed on this EC.
My AWS EBS application seems to failing here when trying to execute this method in GeoDjango:
https://github.com/django/django/blob/86908785076b2bbc31b908781da6b6ad1779b18b/django/contrib/gis/gdal/libgdal.py#L46
Vancity, does your Amazon Linux have libcrypto.so.1.1?
this is very very very slow...
this is very very very slow...
@hanayashiki
It is very dependent on what type of machine you put into play. I'd recommend not going less than a T3.small - if you run a micro or nano it will take forever!
@seanwhudson
I actually encountered errors myself after deployment. I'm not sure about your current instance, but in your last note creating the simlink for
libgdal.so.20did you have a typo? The OSError looks like you want to link tolibgdal.so.20notlibgdal.so.1.I have found a new solution after starting from scratch by modifying install and compile scripts I found at https://gist.github.com/hervenivon/fe3a327bc28b142e51beb38ef11844c0.
[1] Create a script
install-gdal.sh:#!/bin/bash # Installation of GDAL and dependencies. # Based on https://gist.github.com/hervenivon/fe3a327bc28b142e51beb38ef11844c0 # Update with your versions. GEOS_VERSION=3.7.1 GDAL_VERSION=2.3.3 PROJ4_VERSION=4.9.3 sudo yum-config-manager --enable epel sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel # Compilation work for geos. mkdir -p "/tmp/geos-${GEOS_VERSION}-build" cd "/tmp/geos-${GEOS_VERSION}-build" curl -o "geos-${GEOS_VERSION}.tar.bz2" \ "http://download.osgeo.org/geos/geos-${GEOS_VERSION}.tar.bz2" \ && bunzip2 "geos-${GEOS_VERSION}.tar.bz2" \ && tar xvf "geos-${GEOS_VERSION}.tar" cd "/tmp/geos-${GEOS_VERSION}-build/geos-${GEOS_VERSION}" ./configure --prefix=/usr/local/geos # Make in parallel with 2x the number of processors. make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \ && sudo make install \ && sudo ldconfig # Compilation work for proj4. mkdir -p "/tmp/proj-${PROJ4_VERSION}-build" cd "/tmp/proj-${PROJ4_VERSION}-build" curl -o "proj-${PROJ4_VERSION}.tar.gz" \ "http://download.osgeo.org/proj/proj-${PROJ4_VERSION}.tar.gz" \ && tar xfz "proj-${PROJ4_VERSION}.tar.gz" cd "/tmp/proj-${PROJ4_VERSION}-build/proj-${PROJ4_VERSION}" ./configure --prefix=/usr/local/proj4 # Make in parallel with 2x the number of processors. make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \ && sudo make install \ && sudo ldconfig # Compilation work for GDAL mkdir -p "/tmp/gdal-${GDAL_VERSION}-build" cd "/tmp/gdal-${GDAL_VERSION}-build" curl -o "gdal-${GDAL_VERSION}.tar.gz" \ "http://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION}.tar.gz" \ && tar xfz "gdal-${GDAL_VERSION}.tar.gz" cd "/tmp/gdal-${GDAL_VERSION}-build/gdal-${GDAL_VERSION}" ./configure --prefix=/usr/local/gdal \ --with-static-proj4=/usr/local/proj4 \ --without-python # Make in parallel with 2x the number of processors. make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \ && sudo make install \ && sudo ldconfig # Bundle resources. cd /usr/local/geos tar zcvf "/tmp/geos-${GEOS_VERSION}.tar.gz" * cd /usr/local/proj4 tar zcvf "/tmp/proj4-${PROJ4_VERSION}.tar.gz" * cd /usr/local/gdal tar zcvf "/tmp/gdal-${GDAL_VERSION}.tar.gz" *[2] Run the script on an EC2 instance (takes a long time) after which you should have the three bundled files available:
/tmp/geos-3.7.1.tar.gz /tmp/proj4-4.9.3.tar.gz /tmp/gdal-2.3.3.tar.gz[3] Upload these to your S3 bucket.
[4] Add the following to your
.ebextensions. Each command tests for the existence of the directory on your instance and downloads and unzips the source as needed.commands: 01_install_geos: command: | sudo wget [s3-url]/geos-3.7.1.tar.gz sudo mkdir -p /usr/local/geos sudo tar -xvf geos-3.7.1.tar.gz -C /usr/local/geos sudo rm -f geos-3.7.1.tar.gz test: "[ ! -d /usr/local/geos ]" 02_install_proj4: command: | sudo wget [s3-url]/proj4-4.9.3.tar.gz sudo mkdir -p /usr/local/proj4 sudo tar -xvf proj4-4.9.3.tar.gz -C /usr/local/proj4 sudo rm -f proj4-4.9.3.tar.gz test: "[ ! -d /usr/local/proj4 ]" 03_install_gdal: command: | sudo wget [s3-url]/gdal-2.3.3.tar.gz sudo mkdir -p /usr/local/gdal sudo tar -xvf gdal-2.3.3.tar.gz -C /usr/local/gdal sudo rm -f gdal-2.3.3.tar.gz test: "[ ! -d /usr/local/gdal ]"[5] Finally, add the following to your
.ebextensions/options.configfile:option_settings: aws:elasticbeanstalk:application:environment: PATH: /usr/local/gdal/bin:$PATH LD_LIBRARY_PATH: /usr/local/proj4/lib:/usr/local/gdal/lib:$LD_LIBRARY_PATH GDAL_DATA: /usr/local/gdal/share/gdalSeems like a lot of steps now that I've typed it all out but it has been working reliably for me on EB running
Python 3.6 running on 64bit Amazon Linux/2.9.4. Seems much cleaner than bundling the entire/usr/localdirectory too.
Hey,
I am using your solution with Amazon Linux 2 and Python 3.7 but facing issue in deployment. Can you please help for Amazon Linux 2
@rahulkhairnarr What is the issue/can you share error messages? Also, facing issues on EB deployment?
this is very very very slow...
@hanayashiki
It is very dependent on what type of machine you put into play. I'd recommend not going less than a T3.small - if you run a micro or nano it will take forever!
Not forever, just a night's sleep
It worked for me with gdal v2.2.0, django 2.1
I had to change the GDAL path: GDAL_LIBRARY_PATH = /usr/local/lib/libgdal.so in settings.py
Attempting install on a sagemaker notebook instance, make -j4 completes fine but the following sudo make installresults in a massive volume of errors like:
/opt/conda/include/unicode/localpointer.h:224:5: warning: identifier ‘noexcept’ is a keyword in C++11 [-Wc++0x-compat]
LocalPointer(LocalPointer<T> &&src) U_NOEXCEPT : LocalPointerBase<T>(src.ptr) {
Hello, thank you very much for the information I wanted to ask a favor if it is possible to share these 3 files to do a test with sagemaker please
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@seanwhudson
I actually encountered errors myself after deployment. I'm not sure about your current instance, but in your last note creating the simlink for
libgdal.so.20did you have a typo? The OSError looks like you want to link tolibgdal.so.20notlibgdal.so.1.I have found a new solution after starting from scratch by modifying install and compile scripts I found at https://gist.github.com/hervenivon/fe3a327bc28b142e51beb38ef11844c0.
[1] Create a script
install-gdal.sh:[2] Run the script on an EC2 instance (takes a long time) after which you should have the three bundled files available:
[3] Upload these to your S3 bucket.
[4] Add the following to your
.ebextensions. Each command tests for the existence of the directory on your instance and downloads and unzips the source as needed.[5] Finally, add the following to your
.ebextensions/options.configfile:Seems like a lot of steps now that I've typed it all out but it has been working reliably for me on EB running
Python 3.6 running on 64bit Amazon Linux/2.9.4. Seems much cleaner than bundling the entire/usr/localdirectory too.