Skip to content

Instantly share code, notes, and snippets.

@abelcallejo
Last active October 29, 2024 01:29
Show Gist options
  • Save abelcallejo/e75eb93d73db6f163b076d0232fc7d7e to your computer and use it in GitHub Desktop.
Save abelcallejo/e75eb93d73db6f163b076d0232fc7d7e to your computer and use it in GitHub Desktop.
Installing GDAL 3.2.1 on Amazon Linux 2

Installing GDAL 3.2.1 on Amazon Linux 2

gdal linux yum

As of this day, this is probably the only and fastest way of installing it.

Package requirements

Based from the GDAL and PROJ build requirements, here is the full list of required packages to install:

  • C++11 - cpp version 7.3.1
  • PROJ 6 - proj version 6.1.1
    • SQLite3 - sqlite3 version 3.7.17
    • libtiff - version 4.0
    • cmake - version 3.13.1

Installation of these packages are covered in the sections below. However, before you proceed with the installation, make sure you are already logged in to your EC2 Amazon Linux 2 via SSH. After logging in, you just need to issue some statements in the command-line.

1. Installation of required packages

Luckilly, we don't have to install everything from source. At least some can be installed through yum.

sudo yum install gcc-c++.x86_64 cpp.x86_64 sqlite-devel.x86_64 libtiff.x86_64 cmake3.x86_64 -y

2. Installation of PROJ

Installation from source, currently there's no other way but this.

cd /tmp
wget https://download.osgeo.org/proj/proj-6.1.1.tar.gz
tar -xvf proj-6.1.1.tar.gz
cd proj-6.1.1
./configure
sudo make
sudo make install

Important

Be careful not to install PROJ version 6.2.0 or newer because the one from the yum packages is only SQLite version 3.7.17. Installing PROJ version 6.2.0 would require installing SQLite version 3.11 which you may need to install from source and not with yum.

3. Installation of GDAL

Installation from source, currently there's no other way but this.

cd /tmp
wget https://github.com/OSGeo/gdal/releases/download/v3.2.1/gdal-3.2.1.tar.gz
tar -xvf gdal-3.2.1.tar.gz
cd gdal-3.2.1
./configure --with-proj=/usr/local --with-python
sudo make
sudo make install

Testing your installation

which gdalinfo; gdalinfo --version

It should show something like

/usr/local/bin/gdalinfo
GDAL 3.2.1, released 2020/12/29

finally

Comments and suggestions

Did I miss something? Are you having some troubles with the installed GDAL? Write them down as comments below. Let's help our selves and others at the same time.

@Dave-Allured
Copy link

Difficult question -- a simple answer is that it sort of depends on the context, or the intended audience. I think GDAL developers eventually resorted to providing a simple base package with few dependencies and few supported formats, then let people (or repackagers) add on what they want.

One approach would be to look at what some of the big package managers do, and provide a similar set. Another approach would be to ask on the GDAL developers mailing list, but I suspect you would get a similar roundabout answer like mine. That developers list is very active, and has a lot of user exposure, so you might get some good feedback there.

A down side of supporting lots of formats is a lot of dependencies for your own users to have to install. This is accommodated in part by instructing users how to customize their builds. In reality that is what this thread is about. Pardon my ramblings, I do not have a pat answer for this.

@mahmutbilgen
Copy link

I am getting error while make install;

make[3]: Entering directory /apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts/elastic /bin/sh /apps/tmp/gdal-3.2.1/libtool --mode=compile --silent --tag=CXX g++ -std=c++11 -I/apps/tmp/gdal-3.2.1/port -I/apps/tmp/gdal-3.2.1/gcore -I/apps/tmp/gdal-3.2.1/alg -I/apps/tmp/gdal-3.2.1/ogr -I/apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts -I/apps/tmp/gdal-3.2.1/gnm -I/apps/tmp/gdal-3.2.1/apps -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -g -O2 -Wall -Wextra -Winit-self -Wunused-parameter -Wformat -Werror=format-security -Wno-format-nonliteral -Wlogical-op -Wshadow -Werror=vla -Wmissing-declarations -Wnon-virtual-dtor -Woverloaded-virtual -fno-operator-names -Wzero-as-null-pointer-constant -I/apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts/geojson/libjson -I.. -I../.. -I../geojson -DGNM_ENABLED -I/apps/tmp/gdal-3.2.1/port -DGDAL_COMPILATION -c -o ../o/ogrelasticlayer.lo ogrelasticlayer.cpp In file included from ogr_elastic.h:33:0, from ogrelasticlayer.cpp:30: /apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts/ogrsf_frmts.h: In lambda function: /apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts/ogrsf_frmts.h:109:18: error: ‘OGRErr OGRLayer::GetExtentInternal(int, OGREnvelope*, int)’ is protected OGRErr GetExtentInternal(int iGeomField, OGREnvelope *psExtent, int bForce ); ^ ogrelasticlayer.cpp:3513:83: error: within this context const auto eRet = OGRLayer::GetExtentInternal(iGeomField, psExtent, bForce); ^ make[3]: *** [../o/ogrelasticlayer.lo] Error 1 make[3]: Leaving directory /apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts/elastic' make[2]: *** [elastic-target] Error 2 make[2]: Leaving directory /apps/tmp/gdal-3.2.1/ogr/ogrsf_frmts' make[1]: *** [sublibs] Error 2 make[1]: Leaving directory /apps/tmp/gdal-3.2.1/ogr' make: *** [ogr-target] Error 2

@RichardScottOZ
Copy link

another error:-

make[2]: Entering directory `/tmp/gdal-3.2.1/frmts/mbtiles'
/bin/sh /tmp/gdal-3.2.1/libtool --mode=compile --silent --tag=CXX /home/ec2-user/anaconda3/envs/tensorflow_p37/bin/x86_64-conda-linux-gnu-c++ -I/tmp/gdal-3.2.1/port -I/tmp/gdal-3.2.1/gcore -I/tmp/gdal-3.2.1/alg -I/tmp/gdal-3.2.1/ogr -I/tmp/gdal-3.2.1/ogr/ogrsf_frmts -I/tmp/gdal-3.2.1/gnm -I/tmp/gdal-3.2.1/apps -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/ec2-user/anaconda3/envs/tensorflow_p37/include  -Wall -Wextra -Winit-self -Wunused-parameter -Wformat -Werror=format-security -Wno-format-nonliteral -Wlogical-op -Wshadow -Werror=vla -Wdate-time -Wnull-dereference -Wduplicated-cond -Wextra-semi -Wfloat-conversion -Wmissing-declarations -Wnon-virtual-dtor -Woverloaded-virtual -fno-operator-names -Wzero-as-null-pointer-constant -Wsuggest-override -Wimplicit-fallthrough  -I../../ogr/ogrsf_frmts/geojson -I../../ogr/ogrsf_frmts/mvt -I/tmp/gdal-3.2.1/ogr/ogrsf_frmts/geojson/libjson   -DGNM_ENABLED -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/ec2-user/anaconda3/envs/tensorflow_p37/include -I/tmp/gdal-3.2.1/port  -DGDAL_COMPILATION -I../../ogr -I../../ogr/ogrsf_frmts/gpkg -I../../ogr/ogrsf_frmts/sqlite -I/usr/include -DHAVE_SQLITE -c -o ../o/mbtilesdataset.lo mbtilesdataset.cpp
In file included from /home/ec2-user/anaconda3/envs/tensorflow_p37/lib/gcc/x86_64-conda-linux-gnu/9.3.0/include/xmmintrin.h:34,
                 from /home/ec2-user/anaconda3/envs/tensorflow_p37/lib/gcc/x86_64-conda-linux-gnu/9.3.0/include/immintrin.h:29,
                 from /home/ec2-user/anaconda3/envs/tensorflow_p37/lib/gcc/x86_64-conda-linux-gnu/9.3.0/include/x86intrin.h:32,
                 from /tmp/gdal-3.2.1/port/cpl_port.h:716,
                 from /tmp/gdal-3.2.1/gcore/gdal_frmts.h:34,
                 from mbtilesdataset.cpp:34:
/home/ec2-user/anaconda3/envs/tensorflow_p37/lib/gcc/x86_64-conda-linux-gnu/9.3.0/include/mm_malloc.h:34:16: error: declaration of 'int posix_memalign(void**, size_t, size_t)' has a different exception specifier
   34 | extern "C" int posix_memalign (void **, size_t, size_t);
      |                ^~~~~~~~~~~~~~
In file included from /tmp/gdal-3.2.1/port/cpl_port.h:138,
                 from /tmp/gdal-3.2.1/gcore/gdal_frmts.h:34,
                 from mbtilesdataset.cpp:34:
/usr/include/stdlib.h:462:12: note: from previous declaration 'int posix_memalign(void**, size_t, size_t) noexcept'
  462 | extern int posix_memalign (void **__memptr, size_t __alignment, size_t __size)
      |            ^~~~~~~~~~~~~~
In file included from /usr/include/features.h:423,
                 from /usr/include/bits/libc-header-start.h:33,
                 from /usr/include/stdio.h:27,
                 from /tmp/gdal-3.2.1/port/cpl_port.h:137,
                 from /tmp/gdal-3.2.1/gcore/gdal_frmts.h:34,
                 from mbtilesdataset.cpp:34:
/usr/include/sys/cdefs.h:289:63: warning: 'int stat(const char*, stat*)' hides constructor for 'struct stat' [-Wshadow]
  289 | # define __nonnull(params) __attribute__ ((__nonnull__ params))
      |                                                               ^
/usr/include/sys/stat.h:206:43: note: in expansion of macro '__nonnull'
  206 |    struct stat *__restrict __buf) __THROW __nonnull ((1, 2));
      |                                           ^~~~~~~~~
/usr/include/sys/cdefs.h:289:63: warning: 'int stat64(const char*, stat64*)' hides constructor for 'struct stat64' [-Wshadow]
  289 | # define __nonnull(params) __attribute__ ((__nonnull__ params))
      |                                                               ^
/usr/include/sys/stat.h:225:47: note: in expansion of macro '__nonnull'
  225 |      struct stat64 *__restrict __buf) __THROW __nonnull ((1, 2));
      |                                               ^~~~~~~~~
make[2]: *** [../o/mbtilesdataset.lo] Error 1
make[2]: Leaving directory `/tmp/gdal-3.2.1/frmts/mbtiles'
make[1]: *** [mbtiles-install-obj] Error 2
make[1]: Leaving directory `/tmp/gdal-3.2.1/frmts'
make: *** [frmts-target] Error 2

@andrew-plowright
Copy link

Awesome. This worked, although it can't find gdalinfo unless I specify where to find it (ex.: /usr/local/bin/gdalinfo --version)

Any advice on how to install the Python bindings from here?

@RichardScottOZ
Copy link

I think mine might be a problem with how the Deep Learning AMI is set up - as same problems for Ubuntu.

@ababaian
Copy link

Thanks for the code. I needed to install this for some R packages to work, sf, rnaturalearth, and leaflet and had to amend the code to add the libraries to /usr/lib64 for the packages to find them and also installed geos from source. Code is wrapped with && for use in a dockerfile.

  yum install -y gcc-c++.x86_64 cpp.x86_64 sqlite-devel.x86_64 libtiff.x86_64 cmake3.x86_64 &&\
  wget https://download.osgeo.org/geos/geos-3.9.1.tar.bz2 &&\
  tar -xvf geos-3.9.1.tar.bz2 &&\
  cd geos-3.9.1 &&\
  ./configure --libdir=/usr/lib64 &&\
  sudo make &&\
  sudo make install &&\
  wget https://download.osgeo.org/proj/proj-6.1.1.tar.gz &&\
  tar -xvf proj-6.1.1.tar.gz &&\
  cd proj-6.1.1 &&\
  ./configure --libdir=/usr/lib64 &&\
  sudo make &&\
  sudo make install &&\
  cd .. && rm -rf proj-* &&\
  wget https://github.com/OSGeo/gdal/releases/download/v3.2.1/gdal-3.2.1.tar.gz &&\
  tar -xvf gdal-3.2.1.tar.gz &&\
  cd gdal-3.2.1 &&\
  ./configure --libdir=/usr/lib64 --with-proj=/usr/local --with-geos=/usr/local &&\
  sudo make &&\
  sudo make install &&\
  cd .. && rm -rf gdal-* &&\

@abelcallejo
Copy link
Author

@mahmutbilgen @RichardScottOZ You may also try installing newer versions of GDAL. Essentially doing steps 1 & 2 as it is and then opt for a newer GDAL version on step 3.

@phish3y
Copy link

phish3y commented Nov 8, 2021

Thank you!

@ArtemShtephan
Copy link

Thank you!

@RichardScottOZ
Copy link

Yes, if can get the development libraries to support for that particular repository, or have to add that too.

@KHwong12
Copy link

Thanks so much for the guideline! Though, I ran into some error when trying to make gdal, stating that fatal error: Python.h: No such file or directory. Turns out python-devel and python3-devel also required.

@DervinsMaswer
Copy link

i have this error
install: .libs/libgdal.lai does not exist

@russellhoff
Copy link

This is a elastic beanstalk configuration file. Added within .ebextensions as 01-gdal.config:

commands:
  01_install_gdal:
    test: "[ ! -d /usr/local/gdal ]"
    command: "/tmp/gdal_install.sh"
files:
  "/tmp/gdal_install.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      sudo yum -y update
      sudo yum -y upgrade
      sudo amazon-linux-extras install epel -y
      sudo yum -y install gcc-c++ cpp sqlite-devel libtiff cmake3
      cd /tmp
      wget https://download.osgeo.org/proj/proj-6.1.1.tar.gz
      tar -xvf proj-6.1.1.tar.gz
      cd proj-6.1.1
      ./configure
      sudo make
      sudo make install
      cd /tmp
      wget https://github.com/OSGeo/gdal/releases/download/v3.2.1/gdal-3.2.1.tar.gz
      tar -xvf gdal-3.2.1.tar.gz
      cd gdal-3.2.1
      ./configure --with-proj=/usr/local --with-python
      sudo make
      sudo make install
      which gdalinfo; gdalinfo --version

@FilipKubalaThey
Copy link

@russellhoff any help?

[ERROR] Unhandled exception during build: Command 01_install_gdal failed Traceback (most recent call last): File "/opt/aws/bin/cfn-init", line 176, in <module> worklog.build(metadata, configSets) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 135, in build Contractor(metadata).build(configSets, self) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 561, in build self.run_config(config, worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 573, in run_config CloudFormationCarpenter(config, self._auth_config).build(worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 273, in build self._config.commands) File "/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py", line 127, in apply raise ToolError(u"Command %s failed" % name) cfnbootstrap.construction_errors.ToolError: Command 01_install_gdal failed

@russellhoff
Copy link

@filipkubala-th-ey IDK the line which fails... Look deeper within the logs

@vrbrito
Copy link

vrbrito commented Mar 3, 2022

For the ones that are using this inside a docker image and want a straightforward solution, I strongly recommend you to simply use this image as a builder and simply copy the files inside your own image. Here the tutorial is creating an AWS Lambda layer, that may not be relevant at all for most of you, but if you copy the first steps changing "lambci/lambda:build-python3.8" image for your own base image, that will work magically.

@FilipKubalaThey
Copy link

@vrbrito consider 2 limitations of using it on lambda:

  1. 500 MB of storage
  2. 15 min of execution time
    I was able to overcome the 500 MB limit by sending the tiles directly to the S3 bucket after generation and then, if needed, read it from there, but unfortunately, I was unable to overcome 15 minutes of execution time, for the 5.5 GB tif input.
    Were you able to overcome this problem? Or were you able to chain the lambdas and parallel the generation?
    Thanks :)

BTW. I was able to overcome all the issues, switching from Amazon Linux 2 to Ubuntu virtual machine - if you have this option, maybe it's the way to go.

@JolanenL
Copy link

JolanenL commented Apr 8, 2022

It worked fine even when replacing with specific newer version "3.4.2" on step 3.

Getting it run on python3 took some trial and error though. Here are the steps as far as I can remember in case anyone attempts the same.

  • Had to also run "yum install python3-devel"
  • Installed specific module version with "pip3 install gdal==3.4.2"
  • Had to append "/usr/lib:/usr/local/lib" to variable LD_LIBRARY_PATH since python could not find the relevant .so files otherwise

@yam-ma
Copy link

yam-ma commented Jun 8, 2022

Hello! Thank you so much for the guidance. But I still have some issues on python3 (I am using python 3.9). Following @JolanenL comment,

  • I run "yum install python3-devel"
  • I did "pip3.9 install gdal==3.2.1" then got errors 'use_2to3_fixers', saying "error in GDAL setup command: use_2to3 is invalid."
  • I appended "/usr/lib:/usr/local/lib" to variable LD_LIBRARY_PAT but got the same errors from 2.

Any suggestions? Thank you so much in advance.

@yam-ma
Copy link

yam-ma commented Jun 8, 2022

Hello again! I wandered for a while, then find
$ export PYTHONPATH="/usr/local/lib/python3.9/site-packages/"
$ python3.9 -c "from osgeo import gdal"
then GDAL worked fine! Thank you so much!

@russellhoff
Copy link

@russellhoff any help?

[ERROR] Unhandled exception during build: Command 01_install_gdal failed Traceback (most recent call last): File "/opt/aws/bin/cfn-init", line 176, in <module> worklog.build(metadata, configSets) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 135, in build Contractor(metadata).build(configSets, self) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 561, in build self.run_config(config, worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 573, in run_config CloudFormationCarpenter(config, self._auth_config).build(worklog) File "/usr/lib/python3.7/site-packages/cfnbootstrap/construction.py", line 273, in build self._config.commands) File "/usr/lib/python3.7/site-packages/cfnbootstrap/command_tool.py", line 127, in apply raise ToolError(u"Command %s failed" % name) cfnbootstrap.construction_errors.ToolError: Command 01_install_gdal failed

Finally I left the idea of using Amazon Elastic Beanstalk and I turned to using AWS ECR and ECS, dockerizing my python/django project. If anybody need help feel free to ask :)

@yam-ma
Copy link

yam-ma commented Jun 9, 2022

Hello again and again... After threw the comment above, I noticed that gdal did not work fine.

from osgeo import gdal
import numpy as np
ds = gdal.Open('cea.tif', gdal.GA_ReadOnly)
ERROR 4: cea.tif: No such file or directory
ds = gdal.Open('/home/yam-ma/cea.tif', gdal.GA_ReadOnly)
a = np.array([ds.GetRasterBand(i + 1).ReadAsArray() for i in range(ds.RasterCount)])
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal_array.py", line 14, in swig_import_helper
return importlib.import_module(mname)
File "/usr/local/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'osgeo._gdal_array'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "", line 1, in
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal.py", line 3832, in ReadAsArray
from osgeo import gdalnumeric
File "/usr/local/lib/python3.9/site-packages/osgeo/gdalnumeric.py", line 2, in
from osgeo.gdal_array import *
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal_array.py", line 17, in
_gdal_array = swig_import_helper()
File "/usr/local/lib/python3.9/site-packages/osgeo/gdal_array.py", line 16, in swig_import_helper
return importlib.import_module('_gdal_array')
File "/usr/local/lib/python3.9/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_gdal_array'

Does anyone know how to solve the issue? Thank you!

@Odysseas640
Copy link

I had to add these to my Dockerfile to get it to work:

RUN pip install "setuptools<58.0.0" (before installing GDAL)
RUN cp /usr/local/lib/libgdal.so.28* /usr/lib64/ (after installing GDAL)

Now I can run "from osgeo import gdal" in a python3 shell.

@diego-gris
Copy link

I needed to install this for some R packages to work, sf, rnaturalearth, and leaflet

@ababaian, How/where did you install the R packages after getting all these dependencies installed? I am trying to run R on EMR but having trouble to install geospatial packages ('sf', 'terra', 'lidR').

@ababaian
Copy link

@diego-gris here's my docker where it's working (last I checked).

https://github.com/ababaian/palmid/blob/main/Dockerfile

@plazmakeks
Copy link

plazmakeks commented Jul 31, 2023

when configuring gdal i had to add the flags

--with-libpng --with-libpng-devel

to make gdal link with libpng and find some png symbols

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment