Installing GDAL 3.2.1 on Amazon Linux 2

README.md

Installing GDAL 3.2.1 on Amazon Linux 2

As of this day, this is probably the only and fastest way of installing it.

Package requirements

Based from the GDAL and PROJ build requirements, here is the full list of required packages to install:

• C++11 - cpp version 7.3.1
• PROJ 6 - proj version 6.1.1
  • SQLite3 - sqlite3 version 3.7.17
  • libtiff - version 4.0
  • cmake - version 3.13.1

Installation of these packages are covered in the sections below. However, before you proceed with the installation, make sure you are already logged in to your EC2 Amazon Linux 2 via SSH. After logging in, you just need to issue some statements in the command-line.

1. Installation of required packages

Luckilly, we don't have to install everything from source. At least some can be installed through yum.

sudo yum install gcc-c++.x86_64 cpp.x86_64 sqlite-devel.x86_64 libtiff.x86_64 cmake3.x86_64 -y

2. Installation of PROJ

Installation from source, currently there's no other way but this.

cd /tmp
wget https://download.osgeo.org/proj/proj-6.1.1.tar.gz
tar -xvf proj-6.1.1.tar.gz
cd proj-6.1.1
./configure
sudo make
sudo make install

Important

Be careful not to install PROJ version 6.2.0 or newer because the one from the yum packages is only SQLite version 3.7.17. Installing PROJ version 6.2.0 would require installing SQLite version 3.11 which you may need to install from source and not with yum.

3. Installation of GDAL

Installation from source, currently there's no other way but this.

cd /tmp
wget https://github.com/OSGeo/gdal/releases/download/v3.2.1/gdal-3.2.1.tar.gz
tar -xvf gdal-3.2.1.tar.gz
cd gdal-3.2.1
./configure --with-proj=/usr/local --with-python
sudo make
sudo make install

Testing your installation

which gdalinfo; gdalinfo --version

It should show something like

/usr/local/bin/gdalinfo
GDAL 3.2.1, released 2020/12/29

Comments and suggestions

Did I miss something? Are you having some troubles with the installed GDAL? Write them down as comments below. Let's help our selves and others at the same time.

@saheelBreezo, recent GDAL versions install only a minimal set of format drivers by default. Netcdf is not one of the default drivers. Here is a generic method to add a driver to the default installation. This is what @data-ux did above, for the OpenJPEG format.

1. Find and install the support library needed for your format. In this case the support library is netcdf. Either binary install or build from source should work.

2. Specify the desired driver(s) in the GDAL configure command. For netcdf format, the added configure option is --with-netcdf. Include the optional =path sub-argument if the format support library is installed in a non-default location. GDAL can support many different drivers in one build, as long as they are all specified in the configure command.

3. After configure, then build, install, and test GDAL according to the rest of the above instructions.

It may be necessary to set PKG_CONFIG_PATH before GDAL configure. Recent netcdf releases include their own netcdf.pc (pkg-config) file, and GDAL configure might be able to find it automatically. I am not sure how to deal with this in the Amazon Linux context, so you might need to look into this. First see if it works without any adjustment for pkg-config.

For more information about optional GDAL drivers, please see the gdal.org website. For more information about configure driver options, please run ./configure --help.

@data-ux @saheelBreezo @Dave-Allured Do you suggest to include NetCDF and OpenJPG drivers in these installation procedures? Are those drivers more common than not? The goal is to help as many as possible.

Difficult question -- a simple answer is that it sort of depends on the context, or the intended audience. I think GDAL developers eventually resorted to providing a simple base package with few dependencies and few supported formats, then let people (or repackagers) add on what they want.

One approach would be to look at what some of the big package managers do, and provide a similar set. Another approach would be to ask on the GDAL developers mailing list, but I suspect you would get a similar roundabout answer like mine. That developers list is very active, and has a lot of user exposure, so you might get some good feedback there.

A down side of supporting lots of formats is a lot of dependencies for your own users to have to install. This is accommodated in part by instructing users how to customize their builds. In reality that is what this thread is about. Pardon my ramblings, I do not have a pat answer for this.

I am getting error while make install;

make[3]: Entering directory /apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts/elastic /bin/sh /apps/tmp/gdal-3.2.1/libtool --mode=compile --silent --tag=CXX g++ -std=c++11 -I/apps/tmp/gdal-3.2.1/port -I/apps/tmp/gdal-3.2.1/gcore -I/apps/tmp/gdal-3.2.1/alg -I/apps/tmp/gdal-3.2.1/ogr -I/apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts -I/apps/tmp/gdal-3.2.1/gnm -I/apps/tmp/gdal-3.2.1/apps -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2 -Wall -Wextra -Winit-self -Wunused-parameter -Wformat -Werror=format-security -Wno-format-nonliteral -Wlogical-op -Wshadow -Werror=vla -Wmissing-declarations -Wnon-virtual-dtor -Woverloaded-virtual -fno-operator-names -Wzero-as-null-pointer-constant -I/apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts/geojson/libjson -I.. -I../.. -I../geojson -DGNM_ENABLED -I/apps/tmp/gdal-3.2.1/port -DGDAL_COMPILATION -c -o ../o/ogrelasticlayer.lo ogrelasticlayer.cpp In file included from ogr_elastic.h:33:0, from ogrelasticlayer.cpp:30: /apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts/ogrsf_frmts.h: In lambda function: /apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts/ogrsf_frmts.h:109:18: error: ‘OGRErr OGRLayer::GetExtentInternal(int, OGREnvelope*, int)’ is protected OGRErr GetExtentInternal(int iGeomField, OGREnvelope *psExtent, int bForce ); ^ ogrelasticlayer.cpp:3513:83: error: within this context const auto eRet = OGRLayer::GetExtentInternal(iGeomField, psExtent, bForce); ^ make[3]: *** [../o/ogrelasticlayer.lo] Error 1 make[3]: Leaving directory /apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts/elastic' make[2]: *** [elastic-target] Error 2 make[2]: Leaving directory /apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts' make[1]: *** [sublibs] Error 2 make[1]: Leaving directory /apps/tmp/gdal-3.2.1/ogr' make: *** [ogr-target] Error 2

another error:-

make[2]: Entering directory `/tmp/gdal-3.2.1/frmts/mbtiles'
/bin/sh /tmp/gdal-3.2.1/libtool --mode=compile --silent --tag=CXX /home/ec2-user/anaconda3/envs/tensorflow_p37/bin/x86_64-conda-linux-gnu-c++ -I/tmp/gdal-3.2.1/port -I/tmp/gdal-3.2.1/gcore -I/tmp/gdal-3.2.1/alg -I/tmp/gdal-3.2.1/ogr -I/tmp/gdal-3.2.1/ogr/ogrsf_frmts -I/tmp/gdal-3.2.1/gnm -I/tmp/gdal-3.2.1/apps -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/ec2-user/anaconda3/envs/tensorflow_p37/include  -Wall -Wextra -Winit-self -Wunused-parameter -Wformat -Werror=format-security -Wno-format-nonliteral -Wlogical-op -Wshadow -Werror=vla -Wdate-time -Wnull-dereference -Wduplicated-cond -Wextra-semi -Wfloat-conversion -Wmissing-declarations -Wnon-virtual-dtor -Woverloaded-virtual -fno-operator-names -Wzero-as-null-pointer-constant -Wsuggest-override -Wimplicit-fallthrough  -I../../ogr/ogrsf_frmts/geojson -I../../ogr/ogrsf_frmts/mvt -I/tmp/gdal-3.2.1/ogr/ogrsf_frmts/geojson/libjson   -DGNM_ENABLED -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/ec2-user/anaconda3/envs/tensorflow_p37/include -I/tmp/gdal-3.2.1/port  -DGDAL_COMPILATION -I../../ogr -I../../ogr/ogrsf_frmts/gpkg -I../../ogr/ogrsf_frmts/sqlite -I/usr/include -DHAVE_SQLITE -c -o ../o/mbtilesdataset.lo mbtilesdataset.cpp
In file included from /home/ec2-user/anaconda3/envs/tensorflow_p37/lib/gcc/x86_64-conda-linux-gnu/9.3.0/include/xmmintrin.h:34,
                 from /home/ec2-user/anaconda3/envs/tensorflow_p37/lib/gcc/x86_64-conda-linux-gnu/9.3.0/include/immintrin.h:29,
                 from /home/ec2-user/anaconda3/envs/tensorflow_p37/lib/gcc/x86_64-conda-linux-gnu/9.3.0/include/x86intrin.h:32,
                 from /tmp/gdal-3.2.1/port/cpl_port.h:716,
                 from /tmp/gdal-3.2.1/gcore/gdal_frmts.h:34,
                 from mbtilesdataset.cpp:34:
/home/ec2-user/anaconda3/envs/tensorflow_p37/lib/gcc/x86_64-conda-linux-gnu/9.3.0/include/mm_malloc.h:34:16: error: declaration of 'int posix_memalign(void**, size_t, size_t)' has a different exception specifier
   34 | extern "C" int posix_memalign (void **, size_t, size_t);
      |                ^~~~~~~~~~~~~~
In file included from /tmp/gdal-3.2.1/port/cpl_port.h:138,
                 from /tmp/gdal-3.2.1/gcore/gdal_frmts.h:34,
                 from mbtilesdataset.cpp:34:
/usr/include/stdlib.h:462:12: note: from previous declaration 'int posix_memalign(void**, size_t, size_t) noexcept'
  462 | extern int posix_memalign (void **__memptr, size_t __alignment, size_t __size)
      |            ^~~~~~~~~~~~~~
In file included from /usr/include/features.h:423,
                 from /usr/include/bits/libc-header-start.h:33,
                 from /usr/include/stdio.h:27,
                 from /tmp/gdal-3.2.1/port/cpl_port.h:137,
                 from /tmp/gdal-3.2.1/gcore/gdal_frmts.h:34,
                 from mbtilesdataset.cpp:34:
/usr/include/sys/cdefs.h:289:63: warning: 'int stat(const char*, stat*)' hides constructor for 'struct stat' [-Wshadow]
  289 | # define __nonnull(params) __attribute__ ((__nonnull__ params))
      |                                                               ^
/usr/include/sys/stat.h:206:43: note: in expansion of macro '__nonnull'
  206 |    struct stat *__restrict __buf) __THROW __nonnull ((1, 2));
      |                                           ^~~~~~~~~
/usr/include/sys/cdefs.h:289:63: warning: 'int stat64(const char*, stat64*)' hides constructor for 'struct stat64' [-Wshadow]
  289 | # define __nonnull(params) __attribute__ ((__nonnull__ params))
      |                                                               ^
/usr/include/sys/stat.h:225:47: note: in expansion of macro '__nonnull'
  225 |      struct stat64 *__restrict __buf) __THROW __nonnull ((1, 2));
      |                                               ^~~~~~~~~
make[2]: *** [../o/mbtilesdataset.lo] Error 1
make[2]: Leaving directory `/tmp/gdal-3.2.1/frmts/mbtiles'
make[1]: *** [mbtiles-install-obj] Error 2
make[1]: Leaving directory `/tmp/gdal-3.2.1/frmts'
make: *** [frmts-target] Error 2

Awesome. This worked, although it can't find gdalinfo unless I specify where to find it (ex.: /usr/local/bin/gdalinfo --version)

Any advice on how to install the Python bindings from here?

I think mine might be a problem with how the Deep Learning AMI is set up - as same problems for Ubuntu.

Thanks for the code. I needed to install this for some R packages to work, sf, rnaturalearth, and leaflet and had to amend the code to add the libraries to /usr/lib64 for the packages to find them and also installed geos from source. Code is wrapped with && for use in a dockerfile.

  yum install -y gcc-c++.x86_64 cpp.x86_64 sqlite-devel.x86_64 libtiff.x86_64 cmake3.x86_64 &&\
  wget https://download.osgeo.org/geos/geos-3.9.1.tar.bz2 &&\
  tar -xvf geos-3.9.1.tar.bz2 &&\
  cd geos-3.9.1 &&\
  ./configure --libdir=/usr/lib64 &&\
  sudo make &&\
  sudo make install &&\
  wget https://download.osgeo.org/proj/proj-6.1.1.tar.gz &&\
  tar -xvf proj-6.1.1.tar.gz &&\
  cd proj-6.1.1 &&\
  ./configure --libdir=/usr/lib64 &&\
  sudo make &&\
  sudo make install &&\
  cd .. && rm -rf proj-* &&\
  wget https://github.com/OSGeo/gdal/releases/download/v3.2.1/gdal-3.2.1.tar.gz &&\
  tar -xvf gdal-3.2.1.tar.gz &&\
  cd gdal-3.2.1 &&\
  ./configure --libdir=/usr/lib64 --with-proj=/usr/local --with-geos=/usr/local &&\
  sudo make &&\
  sudo make install &&\
  cd .. && rm -rf gdal-* &&\

@mahmutbilgen @RichardScottOZ You may also try installing newer versions of GDAL. Essentially doing steps 1 & 2 as it is and then opt for a newer GDAL version on step 3.

Thank you!

Thank you!

Yes, if can get the development libraries to support for that particular repository, or have to add that too.

Thanks so much for the guideline! Though, I ran into some error when trying to make gdal, stating that fatal error: Python.h: No such file or directory. Turns out python-devel and python3-devel also required.

i have this error
install: .libs/libgdal.lai does not exist

This is a elastic beanstalk configuration file. Added within .ebextensions as 01-gdal.config:

commands:
  01_install_gdal:
    test: "[ ! -d /usr/local/gdal ]"
    command: "/tmp/gdal_install.sh"
files:
  "/tmp/gdal_install.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      sudo yum -y update
      sudo yum -y upgrade
      sudo amazon-linux-extras install epel -y
      sudo yum -y install gcc-c++ cpp sqlite-devel libtiff cmake3
      cd /tmp
      wget https://download.osgeo.org/proj/proj-6.1.1.tar.gz
      tar -xvf proj-6.1.1.tar.gz
      cd proj-6.1.1
      ./configure
      sudo make
      sudo make install
      cd /tmp
      wget https://github.com/OSGeo/gdal/releases/download/v3.2.1/gdal-3.2.1.tar.gz
      tar -xvf gdal-3.2.1.tar.gz
      cd gdal-3.2.1
      ./configure --with-proj=/usr/local --with-python
      sudo make
      sudo make install
      which gdalinfo; gdalinfo --version

@russellhoff any help?

[ERROR] Unhandled exception during build: Command 01_install_gdal failed Traceback (most recent call last): File "/opt/aws/bin/cfn-init", line 176, in <module> worklog.build(metadata, configSets) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 135, in build Contractor(metadata).build(configSets, self) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 561, in build self.run_config(config, worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 573, in run_config CloudFormationCarpenter(config, self._auth_config).build(worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 273, in build self._config.commands) File "/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py", line 127, in apply raise ToolError(u"Command %s failed" % name) cfnbootstrap.construction_errors.ToolError: Command 01_install_gdal failed

@filipkubala-th-ey IDK the line which fails... Look deeper within the logs

For the ones that are using this inside a docker image and want a straightforward solution, I strongly recommend you to simply use this image as a builder and simply copy the files inside your own image. Here the tutorial is creating an AWS Lambda layer, that may not be relevant at all for most of you, but if you copy the first steps changing "lambci/lambda:build-python3.8" image for your own base image, that will work magically.

@vrbrito consider 2 limitations of using it on lambda:

1. 500 MB of storage
2. 15 min of execution time
   I was able to overcome the 500 MB limit by sending the tiles directly to the S3 bucket after generation and then, if needed, read it from there, but unfortunately, I was unable to overcome 15 minutes of execution time, for the 5.5 GB tif input.
   Were you able to overcome this problem? Or were you able to chain the lambdas and parallel the generation?
   Thanks :)

BTW. I was able to overcome all the issues, switching from Amazon Linux 2 to Ubuntu virtual machine - if you have this option, maybe it's the way to go.

It worked fine even when replacing with specific newer version "3.4.2" on step 3.

Getting it run on python3 took some trial and error though. Here are the steps as far as I can remember in case anyone attempts the same.

• Had to also run "yum install python3-devel"
• Installed specific module version with "pip3 install gdal==3.4.2"
• Had to append "/usr/lib:/usr/local/lib" to variable LD_LIBRARY_PATH since python could not find the relevant .so files otherwise

Hello! Thank you so much for the guidance. But I still have some issues on python3 (I am using python 3.9). Following @JolanenL comment,

• I run "yum install python3-devel"
• I did "pip3.9 install gdal==3.2.1" then got errors 'use_2to3_fixers', saying "error in GDAL setup command: use_2to3 is invalid."
• I appended "/usr/lib:/usr/local/lib" to variable LD_LIBRARY_PAT but got the same errors from 2.

Any suggestions? Thank you so much in advance.

Hello again! I wandered for a while, then find
$ export PYTHONPATH="/usr/local/lib/python3.9/site-packages/"
$ python3.9 -c "from osgeo import gdal"
then GDAL worked fine! Thank you so much!

@russellhoff any help?

[ERROR] Unhandled exception during build: Command 01_install_gdal failed Traceback (most recent call last): File "/opt/aws/bin/cfn-init", line 176, in <module> worklog.build(metadata, configSets) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 135, in build Contractor(metadata).build(configSets, self) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 561, in build self.run_config(config, worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 573, in run_config CloudFormationCarpenter(config, self._auth_config).build(worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 273, in build self._config.commands) File "/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py", line 127, in apply raise ToolError(u"Command %s failed" % name) cfnbootstrap.construction_errors.ToolError: Command 01_install_gdal failed

Finally I left the idea of using Amazon Elastic Beanstalk and I turned to using AWS ECR and ECS, dockerizing my python/django project. If anybody need help feel free to ask :)

Hello again and again... After threw the comment above, I noticed that gdal did not work fine.

from osgeo import gdal
import numpy as np
ds = gdal.Open('cea.tif', gdal.GA_ReadOnly)
ERROR 4: cea.tif: No such file or directory
ds = gdal.Open('/home/yam-ma/cea.tif', gdal.GA_ReadOnly)
a = np.array([ds.GetRasterBand(i + 1).ReadAsArray() for i in range(ds.RasterCount)])
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal_array.py", line 14, in swig_import_helper
return importlib.import_module(mname)
File "/usr/local/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'osgeo._gdal_array'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "", line 1, in
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal.py", line 3832, in ReadAsArray
from osgeo import gdalnumeric
File "/usr/local/lib/python3.9/site-packages/osgeo/gdalnumeric.py", line 2, in
from osgeo.gdal_array import *
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal_array.py", line 17, in
_gdal_array = swig_import_helper()
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal_array.py", line 16, in swig_import_helper
return importlib.import_module('_gdal_array')
File "/usr/local/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_gdal_array'

Does anyone know how to solve the issue? Thank you!

I had to add these to my Dockerfile to get it to work:

RUN pip install "setuptools<58.0.0" (before installing GDAL)
RUN cp /usr/local/lib/libgdal.so.28* /usr/lib64/ (after installing GDAL)

Now I can run "from osgeo import gdal" in a python3 shell.

I needed to install this for some R packages to work, sf, rnaturalearth, and leaflet

@ababaian, How/where did you install the R packages after getting all these dependencies installed? I am trying to run R on EMR but having trouble to install geospatial packages ('sf', 'terra', 'lidR').

@diego-gris here's my docker where it's working (last I checked).

https://github.com/ababaian/palmid/blob/main/Dockerfile

when configuring gdal i had to add the flags

--with-libpng --with-libpng-devel

to make gdal link with libpng and find some png symbols