How to install PyGrib with ECCodes (Python 3.6.5, Ubuntu 19, 18, 16, RHEL, Lambda, Linux AMI, Docker)
[ last_updated: 07-JAN-2020 ]
Recently I've had to work with Meterological data of the GRIB format. Needless to say, if you're reading this, you probably know how hard it is to get a library that can read them and even harder, setting up those libraries. Pygrib for python is by far the best library for reading grib files but the installation can leave you dizzy
In this tutorial I use Ubuntu 18 and python 3.6.5 virtualenv. Note: You don't have to use virtualenv Python. You can just as well use your system Python 3 however I personally find virtualenv a clean way to deal with these kind of complicated setups (prevents me mucking up my system's Python). This setup should also work with lower and higher Ubuntu versions (as someone below mentioned) and other Linux distros (like RHEL - AWS Linux AMI etc). The key to this installation is to build eccodes correctly (for your machine architecture) and successfully point pygrib to it. Let me know below if it worked for you with other distros!
But before you go any further, the alternative way to install Pygrib is to use the Conda version (this is by far the easiest way since all the binaries are pre-built): https://anaconda.org/conda-forge/pygrib But it comes with the disadvantage that it is quite large. This is fine but here is the dealbreaker: Since I will be uploading this to AWS lambda, I also need to be able to build Pygrib on either an AWS lambda EC2 (Linux AMI) or AWS Lambda docker image that replicates the AWS Lambda machine architecture. This ensures that the built binaries will work on lambda. Those built for Conda do not work on AWS machine architecture like lambda. I will put the AWS lambda deploy in a separate gist.
UPDATE: User @preeth1 added a very useful dockerfile
for this setup down below in the comments. So if you're really just in a rush, that's another option. If you want to understand the finer details behind the setup, read on.
So without further ado, let's begin!
Ubuntu 18 ships with both Python 2 and 3 but if yours doesn't install Python 3.6.5 by updating aptitude:
sudo apt-get update
sudo apt-get -y upgrade
check the version of python3 installed:
python3 -V
install Python's package manager, pip:
sudo apt-get install -y python3-pip
install Python development tools if you don't have them
sudo apt-get install python3-dev
install python virtualenv
sudo pip3 install virtualenv
Navigate to the directory where you want to create your env and create it, then activate it:
virtualenv <NAME OF YOUR VIRTUAL ENV>
.In my case, 'venv':
virtualenv venv
source venv/bin/activate
you should see something like: (venv) user@computer:
you can deactivate it with $ deactivate anytime to use your system's python
Your virtual Python is now ready. Do you have CMAKE and MAKE though? You'll need them to build eccodes
sudo apt-get install build-essential
sudo apt-get install cmake
First part is done, now time to build eccodes. For more details, see: https://confluence.ecmwf.int/display/ECC/ecCodes+installation
Create a directory where you will build eccodes. I usually put custom builds in one folder called custom_builds in my HOME dir. You can use whatever
mkdir ~/custom_builds/eccodes
cd ~/custom_builds/eccodes
here we will download the eccodes tar file first. Get the link for the latest release here: https://confluence.ecmwf.int//display/ECC/Releases In my case the latest is 2.8.2:
wget https://confluence.ecmwf.int/download/attachments/45757960/eccodes-2.8.2-Source.tar.gz?api=v2
uncompress it:
tar -xzf eccodes-2.8.2-Source.tar.gz?api=v2
create your build directory and cd into it:
mkdir build ; cd build
Now here take a moment to understand the cmake build requirements for eccodes.
From the installation link above, you'll see there are various params that you can enable or disable for any feature you need enabled, like png
, jpg
, fortran
etc. Ensure you have the required dependency installed on your system. In my case, I didn't need any of the extra options. I just needed to read grib files. Take a look at this link: https://jswhit.github.io/pygrib/docs/index.html if you need more information about those extra dependencies.
For example you would need libpng
and the jpeg2000
libs installed on your system to enable png and jpg functionality in pygrib.
I chose ~/source/eccodes
as the location to which I will tell cmake to install eccodes. ( nb: source
is just a folder in my home directory where I install programs. You can choose any location: ~/<path-to-your-preferred-install-location>/eccodes
)
The last line of the cmake command is the folder extracted from the tarball. (\
continues the command on a new line). For ease of readng purposes. While inside the 'build' directory, run:
cmake \
-DCMAKE_INSTALL_PREFIX=~/source/eccodes \
-ENABLE_NETCDF=OFF \
-ENABLE_JPG=OFF \
-ENABLE_PNG=OFF \
-ENABLE_PYTHON=OFF \
-ENABLE_FORTRAN=OFF \
../eccodes-2.8.2-Source
should you have any errors while building, delete the build folder and create it again, then (fix the issues) and run the cmake cmd again:
cd ..; rm -rf build; mkdir build; cd build
Note: If you get an error like -- Configuring incomplete, errors occurred!
your makefile won't be created and you won't be able to run make
in the next step. This could be because the cmake options you configured above (like ENABLE_PNG=OFF
) were not properly loaded. Check the console output and if you see some defaults enabled which you disabled in the cmake command, then it means you're facing the same issue. This happened to me a few times and the only way I solved it was by changing the defaults manually inside the CMakeLists.txt
. To do this (assuming you're still inside the build
directory):
cd ..
cd eccodes-2.8.2-Source
nano CMakeLists.txt
then manually disable or enable whatever it is you need. Just look through the file for DEFAULT
and adjust accordingly. For example, you change the ON
to OFF
for the option below.:
ecbuild_add_option( FEATURE JPG
DESCRIPTION "Support for JPG decoding/encoding"
DEFAULT ON
)
Save the file. Then delete and recreate your build
directory as previously directed then run the cmake
command again (no need to add the options since you've already changed them):
cmake -DCMAKE_INSTALL_PREFIX=~/source/eccodes ../eccodes-2.8.2-Source
It should now run successfully. You should see Configuring done
. You now have a makefile and can continue.
When eccodes is done, run these commands
make
ctest
some of the tests may fail depending on the options (like the disabled dependency tests) you set for cmake, but if most ran fine then eccodes make went okay! Finish the make install with:
make install
you should now have eccodes fine on your machine. You will need to set this environment variable though:
export ECCODES_DEFINITION_PATH= the_path_to_where_you_told_cmake_to_install_eccodes/eccodes/share/eccodes/definitions.
In my case:
nano ~/.bashrc
At the bottom of the file, add:
export ECCODES_DEFINITION_PATH=~/source/eccodes/share/eccodes/definitions
Press CTRL + X
, then ENTER
to save the file. Then source it to set the path in your current terminal session:
source ~/.bashrc
At this point eccodes should be set up A okay! Let's set up pygrib!
I wouldn't recommend installing pygrib from pip (pypi) because,
- it's behind the latest release
- it will likely fail if you changed the default directories for eccodes and its environment variables
I recommend installing pygrib straight from the github:
But first, a few necessary dependencies! pygrib needs numpy
, matplotlib
, basemap
(which needs libgeos
)
Ensure you're still inside your virtualenv (venv). Your console should be showing it like this: (venv) user@computer:
Find the latest version of libgeos
to install. In my case it was 3.6.2
:
sudo apt-get install libgeos-3.6.2 libgeos-dev
pip install numpy matplotlib
pip install https://github.com/matplotlib/basemap/archive/master.zip
Now if everything went well you should be able to install pygrib without any issues. We're going to install pygrib with pip but first we need to edit the setup file.
From any directory. In my case, ~/source
, get the latest pygrib master:
git clone https://github.com/jswhit/pygrib
cd pygrib
We need to create and edit the setup.cfg
file. pygrib provides a template setup.cfg.template
file for us.
renaming the template file:
mv setup.cfg.template setup.cfg
In nano (or your favorite GUI text editor):
nano setup.cfg
In this file, find this line: #grib_api_dir = /usr/local
Uncomment and put the location of eccodes. in my case ~/source/eccodes
(write the full path, in my case /home/emmanuel/source/eccodes
):
grib_api_dir =/home/emmanuel/source/eccodes
In the latest version of eccodes, (>2.0.3), the default is #grib_api_libname = eccodes
But if for some reason you're using (<2.0.3), uncomment the last line and ensure it is: grib_api_libname = eccodes
If you enabled any other dependencies (like jpg and png support) ensure you uncomment those lines to and point them to the install directory of those libraries.
To ensure smooth install, only uncomment what you need (I only have grib_api_dir
uncommented). Everything else is disabled.
Save the file when you're done. In nano, CTRL+X
then ENTER
You can optionally delete the .git
folder. It is useless unless you're going to do a PR contribution on the pygrib project:
sudo rm -rf .git
Now we zip the contents of this folder to get it ready for a pip install:
zip -r ../pygrib.zip *
This places your zip file in the parent directory. Go up one to find it:
cd ..
Now we install it with pip (Are you still inside the venv? Ensure that you are!):
pip install pygrib.zip
If for some reason pip complains and refuses to install, fear not! You can build it and install it yourself:
- copy the unzipped pygrib folder to your virtualenv's site-packages
- again, ensure you're still in the venv:
cp -r pygrib/ ~/venv/lib/python3.6/site-packages/
cd ~/venv/lib/python3.6/site-packages/pygrib
python setup.py build
python setup.py install
python test.py
If all goes well, then you're all set! You have successfully installed pygrib! Happy messing around with the world's meteorological data!
Thanks for this informations.
It remains complicated for me to achive all this.
I would like to use Eccodes in AWS Lambda functions.
So, I need to add a layer in AWS Lambda containing Eccodes and all dependencies.
Could you provide a zip file containing all this as required by AWS Lambda ?
Thanks in advance.