GDAL 2.0 on Amazon Linux

README.md

sudo yum -y update
sudo yum-config-manager --enable epel
sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel
cd /tmp
curl -L http://download.osgeo.org/gdal/2.0.0/gdal-2.0.0.tar.gz | tar zxf -
cd gdal-2.0.0/
./configure --prefix=/usr/local --without-python
make -j4
sudo make install
cd /usr/local
tar zcvf ~/gdal-2.0.0-amz1.tar.gz *

For anyone else I've created a small example repo. Hope it helps!

https://github.com/vancityhuddy/aws-eb-gdal-example

I set up the script and .ebextension/01_options.conf file from the previous posts, but now I am getting the error:

from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/contrib/gis/gdal/libgdal.py", line 47, in <module>
lgdal = CDLL(lib_path)
File "/usr/lib64/python3.6/ctypes/__init__.py", line 343, in __init__
 self._handle = _dlopen(self._name, mode)
`OSError: libcrypto.so.1.1: cannot open shared object file: No such file or directory`

From my understanding, the path to that file is invalid. Here is my 01_options.conf:

...
option_settings:
  aws:elasticbeanstalk:application:environment:
    PATH: /usr/local/gdal/bin:$PATH
    LD_LIBRARY_PATH: /usr/local/proj4/lib:/usr/local/geos/lib:/usr/local/gdal/lib:$LD_LIBRARY_PATH
    GDAL_LIBRARY_PATH: /usr/local/gdal/lib/libgdal.so

Vancityhuddy and essy2017, this is very much appreciated.

I ran into something similar, and iirc it had to do with the gdal versions and/or the output names I was using. Don't go for latest osgeo libraries, I think that caused an issue for me. You'll see in my gdal-build-script from the linked repo that I set the versions to

GEOS_VERSION=3.6.4
GDAL_VERSION=2.4.4
PROJ4_VERSION=5.2.0

A quicker solution (bcs you don't have to recompile) is to verify your lib names. SSH into your ec2 instance, rummage around your /usr/local/ folders or search for which lib filenames exist on the machine. Then you could update your 01_options.conf to match the filenames.

I recompiled with those versions, but no luck.

I tried searching through the EC2 using ssh, but I only found the following:
Location: /usr/lib64/

libcrypto.so                       
libcrypto.so.10         
libcrypto.so.1.0.2k         
libcrypt.so    

Location: /lib64/
libcrypt.so.1

I don't think Libcrypto.so.1.1 is installed on this EC.

My AWS EBS application seems to failing here when trying to execute this method in GeoDjango:
https://github.com/django/django/blob/86908785076b2bbc31b908781da6b6ad1779b18b/django/contrib/gis/gdal/libgdal.py#L46

Vancity, does your Amazon Linux have libcrypto.so.1.1?

this is very very very slow...

this is very very very slow...

@hanayashiki
It is very dependent on what type of machine you put into play. I'd recommend not going less than a T3.small - if you run a micro or nano it will take forever!

@seanwhudson

I actually encountered errors myself after deployment. I'm not sure about your current instance, but in your last note creating the simlink for libgdal.so.20 did you have a typo? The OSError looks like you want to link to libgdal.so.20 not libgdal.so.1.

I have found a new solution after starting from scratch by modifying install and compile scripts I found at https://gist.github.com/hervenivon/fe3a327bc28b142e51beb38ef11844c0.

[1] Create a script install-gdal.sh:

#!/bin/bash

# Installation of GDAL and dependencies.
# Based on https://gist.github.com/hervenivon/fe3a327bc28b142e51beb38ef11844c0

# Update with your versions.
GEOS_VERSION=3.7.1
GDAL_VERSION=2.3.3
PROJ4_VERSION=4.9.3

sudo yum-config-manager --enable epel
sudo yum -y install make automake gcc gcc-c++ libcurl-devel proj-devel geos-devel

# Compilation work for geos.
mkdir -p "/tmp/geos-${GEOS_VERSION}-build"
cd "/tmp/geos-${GEOS_VERSION}-build"
curl -o "geos-${GEOS_VERSION}.tar.bz2" \
    "http://download.osgeo.org/geos/geos-${GEOS_VERSION}.tar.bz2" \
    && bunzip2 "geos-${GEOS_VERSION}.tar.bz2" \
    && tar xvf "geos-${GEOS_VERSION}.tar"
cd "/tmp/geos-${GEOS_VERSION}-build/geos-${GEOS_VERSION}"
./configure --prefix=/usr/local/geos

# Make in parallel with 2x the number of processors.
make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \
 && sudo make install \
 && sudo ldconfig


# Compilation work for proj4.
mkdir -p "/tmp/proj-${PROJ4_VERSION}-build"
cd "/tmp/proj-${PROJ4_VERSION}-build"
curl -o "proj-${PROJ4_VERSION}.tar.gz" \
    "http://download.osgeo.org/proj/proj-${PROJ4_VERSION}.tar.gz" \
    && tar xfz "proj-${PROJ4_VERSION}.tar.gz"
cd "/tmp/proj-${PROJ4_VERSION}-build/proj-${PROJ4_VERSION}"
./configure --prefix=/usr/local/proj4

# Make in parallel with 2x the number of processors.
make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \
 && sudo make install \
 && sudo ldconfig


# Compilation work for GDAL
mkdir -p "/tmp/gdal-${GDAL_VERSION}-build"
cd "/tmp/gdal-${GDAL_VERSION}-build"
curl -o "gdal-${GDAL_VERSION}.tar.gz" \
    "http://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION}.tar.gz" \
    && tar xfz "gdal-${GDAL_VERSION}.tar.gz"
cd "/tmp/gdal-${GDAL_VERSION}-build/gdal-${GDAL_VERSION}"
./configure --prefix=/usr/local/gdal \
            --with-static-proj4=/usr/local/proj4 \
            --without-python

# Make in parallel with 2x the number of processors.
make -j $(( 2 * $(cat /proc/cpuinfo | egrep ^processor | wc -l) )) \
 && sudo make install \
 && sudo ldconfig


# Bundle resources.
cd /usr/local/geos
tar zcvf "/tmp/geos-${GEOS_VERSION}.tar.gz" *

cd /usr/local/proj4
tar zcvf "/tmp/proj4-${PROJ4_VERSION}.tar.gz" *

cd /usr/local/gdal
tar zcvf "/tmp/gdal-${GDAL_VERSION}.tar.gz" *

[2] Run the script on an EC2 instance (takes a long time) after which you should have the three bundled files available:

/tmp/geos-3.7.1.tar.gz
/tmp/proj4-4.9.3.tar.gz
/tmp/gdal-2.3.3.tar.gz

[3] Upload these to your S3 bucket.

[4] Add the following to your .ebextensions. Each command tests for the existence of the directory on your instance and downloads and unzips the source as needed.

commands:

  01_install_geos:
    command: |
      sudo wget [s3-url]/geos-3.7.1.tar.gz
      sudo mkdir -p /usr/local/geos
      sudo tar -xvf geos-3.7.1.tar.gz -C /usr/local/geos
      sudo rm -f geos-3.7.1.tar.gz
    test: "[ ! -d /usr/local/geos ]"

  02_install_proj4:
    command: |
      sudo wget [s3-url]/proj4-4.9.3.tar.gz
      sudo mkdir -p /usr/local/proj4
      sudo tar -xvf proj4-4.9.3.tar.gz -C /usr/local/proj4
      sudo rm -f proj4-4.9.3.tar.gz
    test: "[ ! -d /usr/local/proj4 ]"

  03_install_gdal:
    command: |
      sudo wget [s3-url]/gdal-2.3.3.tar.gz
      sudo mkdir -p /usr/local/gdal
      sudo tar -xvf gdal-2.3.3.tar.gz -C /usr/local/gdal
      sudo rm -f gdal-2.3.3.tar.gz
    test: "[ ! -d /usr/local/gdal ]"

[5] Finally, add the following to your .ebextensions/options.config file:

option_settings:
  aws:elasticbeanstalk:application:environment:
    PATH: /usr/local/gdal/bin:$PATH
    LD_LIBRARY_PATH: /usr/local/proj4/lib:/usr/local/gdal/lib:$LD_LIBRARY_PATH
    GDAL_DATA: /usr/local/gdal/share/gdal

Seems like a lot of steps now that I've typed it all out but it has been working reliably for me on EB running Python 3.6 running on 64bit Amazon Linux/2.9.4. Seems much cleaner than bundling the entire /usr/local directory too.

Hey,

I am using your solution with Amazon Linux 2 and Python 3.7 but facing issue in deployment. Can you please help for Amazon Linux 2

@rahulkhairnarr What is the issue/can you share error messages? Also, facing issues on EB deployment?

this is very very very slow...
@hanayashiki
It is very dependent on what type of machine you put into play. I'd recommend not going less than a T3.small - if you run a micro or nano it will take forever!

Not forever, just a night's sleep

It worked for me with gdal v2.2.0, django 2.1
I had to change the GDAL path: GDAL_LIBRARY_PATH = /usr/local/lib/libgdal.so in settings.py

Attempting install on a sagemaker notebook instance, make -j4 completes fine but the following sudo make installresults in a massive volume of errors like:

/opt/conda/include/unicode/localpointer.h:224:5: warning: identifier ‘noexcept’ is a keyword in C++11 [-Wc++0x-compat]
     LocalPointer(LocalPointer<T> &&src) U_NOEXCEPT : LocalPointerBase<T>(src.ptr) {

Hello, thank you very much for the information I wanted to ask a favor if it is possible to share these 3 files to do a test with sagemaker please