How to install PyGrib with ECCodes (Python 3.6.5, Ubuntu 19, 18, 16, RHEL, Lambda, Linux AMI, Docker)
[ last_updated: 07-JAN-2020 ]
Recently I've had to work with Meterological data of the GRIB format. Needless to say, if you're reading this, you probably know how hard it is to get a library that can read them and even harder, setting up those libraries. Pygrib for python is by far the best library for reading grib files but the installation can leave you dizzy
In this tutorial I use Ubuntu 18 and python 3.6.5 virtualenv. Note: You don't have to use virtualenv Python. You can just as well use your system Python 3 however I personally find virtualenv a clean way to deal with these kind of complicated setups (prevents me mucking up my system's Python). This setup should also work with lower and higher Ubuntu versions (as someone below mentioned) and other Linux distros (like RHEL - AWS Linux AMI etc). The key to this installation is to build eccodes correctly (for your machine architecture) and successfully point pygrib to it. Let me know below if it worked for you with other distros!
But before you go any further, the alternative way to install Pygrib is to use the Conda version (this is by far the easiest way since all the binaries are pre-built): https://anaconda.org/conda-forge/pygrib But it comes with the disadvantage that it is quite large. This is fine but here is the dealbreaker: Since I will be uploading this to AWS lambda, I also need to be able to build Pygrib on either an AWS lambda EC2 (Linux AMI) or AWS Lambda docker image that replicates the AWS Lambda machine architecture. This ensures that the built binaries will work on lambda. Those built for Conda do not work on AWS machine architecture like lambda. I will put the AWS lambda deploy in a separate gist.
UPDATE: User @preeth1 added a very useful dockerfile for this setup down below in the comments. So if you're really just in a rush, that's another option. If you want to understand the finer details behind the setup, read on.
So without further ado, let's begin!
Ubuntu 18 ships with both Python 2 and 3 but if yours doesn't install Python 3.6.5 by updating aptitude:
sudo apt-get update
sudo apt-get -y upgradecheck the version of python3 installed:
python3 -Vinstall Python's package manager, pip:
sudo apt-get install -y python3-pipinstall Python development tools if you don't have them
sudo apt-get install python3-devinstall python virtualenv
sudo pip3 install virtualenv Navigate to the directory where you want to create your env and create it, then activate it:
virtualenv <NAME OF YOUR VIRTUAL ENV> .In my case, 'venv':
virtualenv venv
source venv/bin/activateyou should see something like: (venv) user@computer:
you can deactivate it with $ deactivate anytime to use your system's python
Your virtual Python is now ready. Do you have CMAKE and MAKE though? You'll need them to build eccodes
sudo apt-get install build-essential
sudo apt-get install cmakeFirst part is done, now time to build eccodes. For more details, see: https://confluence.ecmwf.int/display/ECC/ecCodes+installation
Create a directory where you will build eccodes. I usually put custom builds in one folder called custom_builds in my HOME dir. You can use whatever
mkdir ~/custom_builds/eccodes
cd ~/custom_builds/eccodeshere we will download the eccodes tar file first. Get the link for the latest release here: https://confluence.ecmwf.int//display/ECC/Releases In my case the latest is 2.8.2:
wget https://confluence.ecmwf.int/download/attachments/45757960/eccodes-2.8.2-Source.tar.gz?api=v2uncompress it:
tar -xzf eccodes-2.8.2-Source.tar.gz?api=v2create your build directory and cd into it:
mkdir build ; cd buildNow here take a moment to understand the cmake build requirements for eccodes.
From the installation link above, you'll see there are various params that you can enable or disable for any feature you need enabled, like png, jpg, fortran etc. Ensure you have the required dependency installed on your system. In my case, I didn't need any of the extra options. I just needed to read grib files. Take a look at this link: https://jswhit.github.io/pygrib/docs/index.html if you need more information about those extra dependencies.
For example you would need libpng and the jpeg2000 libs installed on your system to enable png and jpg functionality in pygrib.
I chose ~/source/eccodes as the location to which I will tell cmake to install eccodes. ( nb: source is just a folder in my home directory where I install programs. You can choose any location: ~/<path-to-your-preferred-install-location>/eccodes)
The last line of the cmake command is the folder extracted from the tarball. (\ continues the command on a new line). For ease of readng purposes. While inside the 'build' directory, run:
cmake \
-DCMAKE_INSTALL_PREFIX=~/source/eccodes \
-ENABLE_NETCDF=OFF \
-ENABLE_JPG=OFF \
-ENABLE_PNG=OFF \
-ENABLE_PYTHON=OFF \
-ENABLE_FORTRAN=OFF \
../eccodes-2.8.2-Sourceshould you have any errors while building, delete the build folder and create it again, then (fix the issues) and run the cmake cmd again:
cd ..; rm -rf build; mkdir build; cd buildNote: If you get an error like -- Configuring incomplete, errors occurred! your makefile won't be created and you won't be able to run make in the next step. This could be because the cmake options you configured above (like ENABLE_PNG=OFF) were not properly loaded. Check the console output and if you see some defaults enabled which you disabled in the cmake command, then it means you're facing the same issue. This happened to me a few times and the only way I solved it was by changing the defaults manually inside the CMakeLists.txt. To do this (assuming you're still inside the build directory):
cd ..
cd eccodes-2.8.2-Source
nano CMakeLists.txt
then manually disable or enable whatever it is you need. Just look through the file for DEFAULT and adjust accordingly. For example, you change the ON to OFF for the option below.:
ecbuild_add_option( FEATURE JPG
DESCRIPTION "Support for JPG decoding/encoding"
DEFAULT ON
)
Save the file. Then delete and recreate your build directory as previously directed then run the cmake command again (no need to add the options since you've already changed them):
cmake -DCMAKE_INSTALL_PREFIX=~/source/eccodes ../eccodes-2.8.2-SourceIt should now run successfully. You should see Configuring done. You now have a makefile and can continue.
When eccodes is done, run these commands
make
ctestsome of the tests may fail depending on the options (like the disabled dependency tests) you set for cmake, but if most ran fine then eccodes make went okay! Finish the make install with:
make installyou should now have eccodes fine on your machine. You will need to set this environment variable though:
export ECCODES_DEFINITION_PATH= the_path_to_where_you_told_cmake_to_install_eccodes/eccodes/share/eccodes/definitions. In my case:
nano ~/.bashrcAt the bottom of the file, add:
export ECCODES_DEFINITION_PATH=~/source/eccodes/share/eccodes/definitionsPress CTRL + X, then ENTER to save the file. Then source it to set the path in your current terminal session:
source ~/.bashrcAt this point eccodes should be set up A okay! Let's set up pygrib!
I wouldn't recommend installing pygrib from pip (pypi) because,
- it's behind the latest release
- it will likely fail if you changed the default directories for eccodes and its environment variables
I recommend installing pygrib straight from the github:
But first, a few necessary dependencies! pygrib needs numpy, matplotlib, basemap (which needs libgeos)
Ensure you're still inside your virtualenv (venv). Your console should be showing it like this: (venv) user@computer:
Find the latest version of libgeos to install. In my case it was 3.6.2:
sudo apt-get install libgeos-3.6.2 libgeos-dev
pip install numpy matplotlib
pip install https://github.com/matplotlib/basemap/archive/master.zipNow if everything went well you should be able to install pygrib without any issues. We're going to install pygrib with pip but first we need to edit the setup file.
From any directory. In my case, ~/source, get the latest pygrib master:
git clone https://github.com/jswhit/pygrib
cd pygribWe need to create and edit the setup.cfg file. pygrib provides a template setup.cfg.template file for us.
renaming the template file:
mv setup.cfg.template setup.cfgIn nano (or your favorite GUI text editor):
nano setup.cfgIn this file, find this line: #grib_api_dir = /usr/local
Uncomment and put the location of eccodes. in my case ~/source/eccodes (write the full path, in my case /home/emmanuel/source/eccodes):
grib_api_dir =/home/emmanuel/source/eccodesIn the latest version of eccodes, (>2.0.3), the default is #grib_api_libname = eccodes
But if for some reason you're using (<2.0.3), uncomment the last line and ensure it is: grib_api_libname = eccodes
If you enabled any other dependencies (like jpg and png support) ensure you uncomment those lines to and point them to the install directory of those libraries.
To ensure smooth install, only uncomment what you need (I only have grib_api_dir uncommented). Everything else is disabled.
Save the file when you're done. In nano, CTRL+X then ENTER
You can optionally delete the .git folder. It is useless unless you're going to do a PR contribution on the pygrib project:
sudo rm -rf .gitNow we zip the contents of this folder to get it ready for a pip install:
zip -r ../pygrib.zip *This places your zip file in the parent directory. Go up one to find it:
cd ..Now we install it with pip (Are you still inside the venv? Ensure that you are!):
pip install pygrib.zipIf for some reason pip complains and refuses to install, fear not! You can build it and install it yourself:
- copy the unzipped pygrib folder to your virtualenv's site-packages
- again, ensure you're still in the venv:
cp -r pygrib/ ~/venv/lib/python3.6/site-packages/
cd ~/venv/lib/python3.6/site-packages/pygrib
python setup.py build
python setup.py install
python test.py If all goes well, then you're all set! You have successfully installed pygrib! Happy messing around with the world's meteorological data!
@saguiMet
Is the
setup.cfgfile created correctly and pointing to the correct path for eccodes? And if so do you have these two lines in yoursetup.cfg?Can you post the contents of that file? Also, where is your eccodes installed (path)?