Skip to content

Instantly share code, notes, and snippets.

@copernicusmarinegist

copernicusmarinegist/convert-netcdf-to-grib-using-cdo-only-v0.1.0.ipynb

Last active October 18, 2024 22:29
Show Gist options
  • Save copernicusmarinegist/3b69345ef0352817e1ce1b8232e0df61 to your computer and use it in GitHub Desktop.
Save copernicusmarinegist/3b69345ef0352817e1ce1b8232e0df61 to your computer and use it in GitHub Desktop.
Download ZIP
Display the source blob
Display the rendered blob
Raw
convert-netcdf-to-grib-using-cdo-only-v0.1.0.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h1><center> CONVERT NETCDF TO GRIB1 - USING CDO ONLY </center></h1>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"\n",
"# Table of Content\n",
"---\n",
"- [Introduction](#introduction)\n",
" - What are GRIB files?\n",
" - Some limitations\n",
"<br><br>\n",
"- [Prerequisite](#prerequisite)\n",
" - Package Manager\n",
" - NetCDF files and Copernicus\n",
"<br><br>\n",
"- [Climate Data Operators](#climate-data-operator)\n",
" - CDO Official Documentation\n",
" - CDO Frequently Asked Questions\n",
" - CDO Installation (with conda)\n",
"<br><br>\n",
"- [Convertion from NetCDF3 to GRIB1](#convertion-netcdf3-to-grib1)\n",
" - Working-directory\n",
" - Get to know your data (NetCDF3 file content)\n",
" - Command lines to convert to GRIB\n",
" - Get to know your data (freshly created GRIB1 file content)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Introduction <a name=\"introduction\"></a>\n",
"---\n",
"\n",
"In this article we will focus on the data conversion from netCDF to GRIB1, using the Climate Data Operators, a collection of command-line operators, developped by the Max-Planck-Institut für Meteorologie.\n",
"\n",
"## What are GRIB files\n",
"---\n",
"The **GRIB** file format is designed for storing and distributing **weather data**. GRIB files are widely used in meteorological applications ([Wikipedia article](https://en.wikipedia.org/wiki/GRIB), including a list of software to read GRIB files).\n",
"\n",
"GRIB stands for \"General Regularly distributed Information in Binary form\" and is a [WMO](https://www.wmo.int/pages/index_en.html) (World Meteorological Organisation) standard format for archiving and exchanging gridded data. GRIB is a binary format, and the data is packed to increase storage efficiency. GRIB data is also self-describing, meaning that the information needed to read the file is present within the file. GRIB files usually have the extension .grib, .grb or .gb.\n",
"\n",
"Currently there are two different coding standards: **GRIB edition 1** (commonly referred to as [GRIB1](http://www.wmo.int/pages/prog/www/WMOCodes/Guides/GRIB/GRIB1-Contents.html)) and **GRIB edition 2** ([GRIB2](http://www.wmo.int/pages/prog/www/WMOCodes.html)). Note that a GRIB file can contain a mix of editions 1 and 2. Major differences are in the structure of the messages, in GRIB2, several variables are defined with more precision (e.g. in GRIB1, latitudes and longitudes are in milli-degrees while in GRIB2, they are in micro-degrees). Also in GRIB2, longitude values must lie between 0 and 360 degrees), encoding of the parameter is very different and in GRIB2, the description of the data is template/table based. See WMO \"[Introduction to GRIB Edition 1 and GRIB Edition 2](https://www.wmo.int/pages/prog/www/WMOCodes/Guides/GRIB/Introduction_GRIB1-GRIB2.pdf)\" documentation for further details.\n",
"\n",
"## Some limitation\n",
"---\n",
"- Note that due to limitations of the GRIB specifications in relation to Marine/Ocean Field, the GRIB version may contains fewer parameters (variables) and incomplete metadata (information about the data). The GRIB format is not formally supported by the Copernicus Marine Environment and Monitoring Service.\n",
"- For this reason, this article will be focus only on the convertion from NetCDF3 to GRIB1 and is only intended to provide a working example for a Copernicus Marine User. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Prerequisite <a name=\"prerequisite\"></a>\n",
"---\n",
"\n",
"## Package Manager\n",
"---\n",
"Installation of a **Package manager** (**Anaconda** or **Miniconda**) to ease \"our life\" (and use multiplatform command lines)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Get the right installer version for your Operating System from [Miniconda website source](https://conda.io/miniconda.html).\n",
"\n",
"<h5>Note</h5>\n",
"In this tutorial, we are using a CentOS 7 Linux 64bits machine and we will focus on saving space on our user session (limited hard drive space available, and limite rights -- we WON'T use admin rights to run command line)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1 ) Download the [installer](https://conda.io/miniconda.html). Example for Linux 64-bit (to get a Python3 distribution):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cd /home/mercator/dbazin/"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": [
"wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2 ) Install Miniconda3 in the directory of your choice (here it's the current directory)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"sh Miniconda3-latest-Linux-x86_64.sh"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3 ) Follow the prompts on the installer screens.\n",
"\n",
"If you are unsure about any setting, accept the defaults. You can change them later."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4 ) To make the changes take effect, **close** and **then re-open your Terminal window**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"5 ) To test your installation, in your Terminal window, run the command:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"conda list"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For a successful installation, a list of installed packages appears."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h5>Important</h5>\n",
"<br>\n",
"Packages are handled below the home directory (default), so **root-access is not needed** (= any user with even low permission can install it). <br>\n",
"_It supports user defined recipies mainly distributed via binstar (https://binstar.org)._"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The below command will add the **conda-forge channel**, which is a project maintaining a lot of scientific software package for conda."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"conda config --add channels conda-forge"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Copernicus NetCDF Files\n",
"---\n",
"### Introduction\n",
"Copernicus Marine Product *Standard Output* (file format) is **NetCDF (.nc)**. If you aren’t familiar with this data format, the [netCDF FAQ](http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#What-Is-netCDF) is a good place to start.<br>\n",
"### Copernicus Marine Service and .nc files discovery\n",
"Products and Dataset are hosted on MOTU servers (OpenDAP-like) and served in NetCDF3 format. Go to the end ([hyperlink](#discovering-cmems)) of the last section available in this tutorial to get more details.<br>\n",
"### NetCDF Data Extrator Downloader Software - e.g. with `motuclient` application <a name=\"netcdf-data-extractor-downloader-software---e.g.-with-motuclient-application\"></a>\n",
"In this subsection, we will answer the question: how to extract and download Copernicus Marine Products in NetCDF files?<br>\n",
"<h5> Note </h5>\n",
"_Until the end of this subsection, the below command lines (and variables) have been set according to this [article](http://marine.copernicus.eu/faq/what-are-the-motu-and-python-requirements/?idpage=169) available in the [Copernicus Marine HelpCenter](http://marine.copernicus.eu/services-portfolio/technical-faq/)._"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### System setup "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1 ) But at first, we need to set our system in order to use the NetCDF data download software: the `motuclient` app. So, we set a variable `software_dir` whose value is the directory path of our choice."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"software_dir = '/home/mercator/dbazin/software/'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2 ) Then, we set the filename `netcdf_data_dl_app` of the archive file containing the python files of the `motuclient` app (this `.tar.gz` filename is published in this [article](http://marine.copernicus.eu/faq/what-are-the-motu-and-python-requirements/?idpage=169))."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"netcdf_data_dl_app = 'motu-client-python-1.5.00-20180223190259664-bin.tar.gz'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3 ) We retrieve the URL from the [GitHub repository](https://github.com/clstoulouse/motu-client-python) of the `motuclient` app, for the recommended version as of April 2018, which is `v1.5.00`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"netcdf_data_dl_app_URL = 'https://github.com/clstoulouse/motu-client-python/releases/download/motu-client-python-1.5.00/'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4 ) Now, we are able to build the command line to download the files according to our preferences. (using [Wget](), but we could have used [cURL]() or any other tool for transferring data specified by URL)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"wget $netcdf_data_dl_app_URL$netcdf_data_dl_app --directory-prefix=$software_dir"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Installation: motuclient app from .tar.gz"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"5 ) Finally, we extract the archive file `netcdf_data_dl_app`, move and rename its folder to the one of our choice: `motuclient`. It's a dumb way, but it would work pretty much everywhere (= any unlix-like platforms) though."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"tar xvf $software_dir$netcdf_data_dl_app --directory=$software_dir && mv $software_dir'/motu-client-python' $software_dir'motuclient/' && rm -rf $software_dir$netcdf_data_dl_app "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"motuclient_app = software_dir + 'motuclient/motu-client.py'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Testing the installation only compatible with Python2.7"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"6 ) Last but not the least, we can now download a Copernicus Marine Product and request our own/customized dataset, by submitting a \"job data extraction\" (also referred as Data Subsetting) thanks to the `motuclient` app."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h5> Important </h5>\n",
"\n",
"\n",
"The `motuclient` app **needs Python2** to be executed (not yet compatible with Python3). Thus, check at first `whereis python2` on your system."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"whereis python2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If it is not yet installed (few chances), then you'll get an empty answer. In that case, use the below command to create a Python2 environment with conda:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"conda create -n python2 anaconda python=2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"7 ) Check the `motuclient` app version, it should be equal or superior to 1.5.00."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"python2 $motuclient_app --version"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Usage of motuclient app"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"8 ) Read [this article](http://marine.copernicus.eu/faq/how-to-write-and-run-the-script-to-download-cmems-products-through-subset-or-direct-download-mechanisms/?idpage=169) to automatically generate options in order to use `motuclient` app (data extractor and downloader)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"python2 $motuclient_app \\\n",
" --user <USERNAME> --pwd <PASSWORD> \\\n",
" --motu http://nrt.cmems-du.eu/motu-web/Motu \\\n",
" --service-id MEDSEA_ANALYSIS_FORECAST_PHY_006_013-TDS \\\n",
" --product-id sv04-med-ingv-tem-an-fc-d \\\n",
" --latitude-min 35.02 --latitude-max 37.85 \\\n",
" --longitude-min 15. --longitude-max 19.83 \\\n",
" --depth-min 1.0182366371154785 --depth-max 5754.0439453125 \\\n",
" --date-min \"2018-03-18\" --date-max \"2018-03-27\" \\\n",
" --variable sea_water_potential_temperature \\\n",
" --out-dir /home/mercator/dbazin/netcdf-to-grib/data/ \\\n",
" --out-name CMEMS7490_Temperature.nc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<h5>Note</h5>\n",
"_For more details and examples, feel free to check the [Copernicus Helpcenter]() and search for articles related to `data download` , `data subset` and `data extraction`._"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"# Climate Data Operator <a name=\"climate-data-operator\"></a>\n",
"---\n",
"\n",
"### [CDO Official Documentation](https://code.mpimet.mpg.de/projects/cdo/wiki/Cdo#Documentation)\n",
"### [CDO Frequently Asked Questions](https://code.mpimet.mpg.de/projects/cdo/wiki/FAQ)\n",
"### CDO Installation - via Conda"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Not covered in this article:\n",
"- [From source](https://code.mpimet.mpg.de/projects/cdo/embedded/index.html#x1-30001.1)\n",
"- [Linux](https://code.mpimet.mpg.de/projects/cdo/wiki/Linux_Platform)\n",
"- [MacOS](https://code.mpimet.mpg.de/projects/cdo/wiki/MacOS_Platform)\n",
"- [FreeBSD](https://code.mpimet.mpg.de/projects/cdo/wiki/FreeBSD_Platform)\n",
"- [Windows](https://code.mpimet.mpg.de/projects/cdo/wiki/Win32)\n",
"- [Docker](https://code.mpimet.mpg.de/projects/cdo/wiki/Docker)\n",
"<br>\n",
"\n",
"Below is the recommended and step-by-step method explained in this tutorial:\n",
"- MultiPlatform Anaconda - Conda (recommended)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you have successfully followed the above **[Prerequisite](#prerequisite)**, now you can install **cdo** and its **python bindings** with:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"conda install cdo\n",
"conda install python-cdo"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"_If there are packages missing and the cdo binary does not run correctly, please consider runnning a complete update of the packages with:_"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# conda update -all"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Convertion NetCDF3 to GRIB1 <a name=\"convertion-netcdf3-to-grib1\"></a>\n",
"---\n",
"The convertion will be illustrated using a real use case example with Copernicus Marine data downloaded from http://marine.copernicus.eu\n",
"\n",
"## Working directory\n",
"---\n",
"We list the content of the directory containing the NetCDF file to be converted to GRIB1. For the record, this NetCDF file has been downloaded using [this method](#netcdf-data-extractor-downloader-software---e.g.-with-motuclient-application)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"working_dir = '/home/mercator/dbazin/netcdf-to-grib/data/'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cd $working_dir"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": [
"ls $working_dir"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll need a **GRIB Table** which we'll create from source and a directory in which we'll store it.\n",
"\n",
"So we create a specific directory `wrf-gribtable` :"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"gribtable = '/home/mercator/dbazin/netcdf-to-grib/wrf-gribtable/'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"mkdir $gribtable"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Get to know your data (NetCDF3 file content)\n",
"---\n",
"It is always a good idea to always examine your input data and get familiar with it before using the data. "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"netcdf_file = 'CMEMS7490_Temperature.nc'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Metadata Header <a name=\"metadata-header\"></a>\n",
"\n",
"We will use the `sinfon` command line from CDO to display the **Metadata Header**:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo sinfon CMEMS7490_Temperature.nc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"_For curiosity, we'll use the 'ncdump -h' command line from [NCO](http://nco.sourceforge.net/) (use `conda install nco`) to display the **Metadata Header** (we notice that we get more or less the same description, though NCO ouput is more detailed):_"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"ncdump -h CMEMS7490_Temperature.nc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Check the NetCDF file version"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": [
"ncdump -k CMEMS7490_Temperature.nc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Check the variables values\n",
"- depth\n",
"- time\n",
"- sea water temperature (skipped)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"ncdump -v depth CMEMS7490_Temperature.nc"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"ncdump -v time -t CMEMS7490_Temperature.nc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Conclusion about NetCDF content\n",
"\n",
"The `netcdf_file` contains **Potential Sea Water Temperature (3D)** whose temporal resolution is **Daily Mean**.\n",
"\n",
"The **3 Dimensions** are:\n",
"- Time: From 2018-03-18 to 2018-03-27\n",
"- Depth: 141 levels from surface (-1.018237 m) to bottom (-5 7541.044 m)\n",
"- BoundingBox: latitude_min = 35.02, latitude_max = 37.85, longitude_min = 15., longitude_max = 19.83"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### WARNING : Degree Celcius (C°) and Kelvin (K)\n",
"As we can see from the **Metadata header**, the Sea Water Temperature units is:\n",
"\n",
" thetao:units = \"degC\" ;\n",
"\n",
"And we know that some **GRIB readers/software** may work **only with Kelvin**. \n",
"\n",
"Thus we need to convert these values (from C° to K) and we will do it using two operators (`addc` and `-setattribute`)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo addc,+273.15 CMEMS7490_Temperature.nc CMEMS7490_Temperature_Kelvin_new.nc"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo -setattribute,thetao@units=\"K\" CMEMS7490_Temperature_Kelvin_new.nc CMEMS7490_Temperature_Kelvin.nc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Command lines to convert to GRIB\n",
"---\n",
"In this part, each command line is described with a use case example about Sea Surface Temperature. It can be extended to waves, currents (u and v), salinity and mixed layer depth, among others."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prerequisite: GRIB Table creation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This convertion example is based on the [WORLD METEOROLOGICAL ORGANIZATION website](http://www.wmo.int/pages/prog/www/WMOCodes.html) and its Manual on Codes, Volume I.2, 2015 Edition updated in 2016, which ban be find [here](http://www.wmo.int/pages/prog/www/WMOCodes/WMO306_vI2/PrevEDITIONS/GRIB1/GRIB1.html)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<table border=\"1\" cellpadding=\"5\" cellspacing=\"0\">\n",
" <tbody>\n",
" <tr>\n",
" <th>VALUE</th>\n",
" <th>PARAMETER</th>\n",
" <th>UNITS</th>\n",
" <th>ABBREV.</th>\n",
" </tr>\n",
" <tr>\n",
" <td>000</td>\n",
" <td>Reserved</td>\n",
" <td>\n",
" <center><br></center>\n",
" </td>\n",
" <td>\n",
" <center><br></center>\n",
" </td>\n",
" </tr>\n",
" <tr>\n",
" <td>049</td>\n",
" <td>u-component of current</td>\n",
" <td>m/s</td>\n",
" <td>UOGRD</td>\n",
" </tr>\n",
" <tr>\n",
" <td>050</td>\n",
" <td>v-component of current</td>\n",
" <td>m/s</td>\n",
" <td>VOGRD</td>\n",
" </tr>\n",
" <tr>\n",
" <td>067</td>\n",
" <td>Mixed layer depth</td>\n",
" <td>m</td>\n",
" <td>MIXHT</td>\n",
" </tr>\n",
" <tr>\n",
" <td>080</td>\n",
" <td>Water Temperature</td>\n",
" <td>K</td>\n",
" <td>WTMP</td>\n",
" </tr>\n",
" <tr>\n",
" <td>088</td>\n",
" <td>Salinity</td>\n",
" <td>kg/kg</td>\n",
" <td>SALTY</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%HTML\n",
"<table border=\"1\" cellpadding=\"5\" cellspacing=\"0\">\n",
" <tbody>\n",
" <tr>\n",
" <th>VALUE</th>\n",
" <th>PARAMETER</th>\n",
" <th>UNITS</th>\n",
" <th>ABBREV.</th>\n",
" </tr>\n",
" <tr>\n",
" <td>000</td>\n",
" <td>Reserved</td>\n",
" <td>\n",
" <center><br></center>\n",
" </td>\n",
" <td>\n",
" <center><br></center>\n",
" </td>\n",
" </tr>\n",
" <tr>\n",
" <td>049</td>\n",
" <td>u-component of current</td>\n",
" <td>m/s</td>\n",
" <td>UOGRD</td>\n",
" </tr>\n",
" <tr>\n",
" <td>050</td>\n",
" <td>v-component of current</td>\n",
" <td>m/s</td>\n",
" <td>VOGRD</td>\n",
" </tr>\n",
" <tr>\n",
" <td>067</td>\n",
" <td>Mixed layer depth</td>\n",
" <td>m</td>\n",
" <td>MIXHT</td>\n",
" </tr>\n",
" <tr>\n",
" <td>080</td>\n",
" <td>Water Temperature</td>\n",
" <td>K</td>\n",
" <td>WTMP</td>\n",
" </tr>\n",
" <tr>\n",
" <td>088</td>\n",
" <td>Salinity</td>\n",
" <td>kg/kg</td>\n",
" <td>SALTY</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create the `GRIB1_table.txt` file with the following content:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"GRIB1_table_content = \"080 Water Temperature K\"\n",
"GRIB1_table = gribtable + 'GRIB1_Table.txt'"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"with open((GRIB1_table), 'a') as f:\n",
" f.write(GRIB1_table_content)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cat $GRIB1_table"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Prepare NetCDF File\n",
"\n",
"Extract the upmost level (`z_min` value from **Metadata header**, [see above](#metadata-header)) in a new netCDF file named `thetao_med_sea_surface.tmp.nc` :"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo -sellevel,1.01823663711548 -selname,thetao CMEMS7490_Temperature_Kelvin.nc thetao_med_sea_surface.tmp.nc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Name the variable (`sea_water_temperature` aka `thetao` which is `var80` from GRIB convention standard) with the appropriate convention standard name from [GRIB1 Table](http://www.wmo.int/pages/prog/www/WMOCodes/WMO306_vI2/PrevEDITIONS/GRIB1/GRIB1.html)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo -setname,var80 thetao_med_sea_surface.tmp.nc thetao_med_sea_surface.tmp2.nc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's focus on sea surface level, so level 0 is needed"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo -setlevel,0 thetao_med_sea_surface.tmp2.nc thetao_med_sea_surface.tmp3.nc"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### GRIB1 creation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Create the GRIB1 file (`thetao_med_sea_surface.tmp1.grb`) with the mandatory metadata from GRIB reader to make sure it's well ingested, by specifying the `GRIB1_Table` to be used. \n",
"\n",
"Further code parameters will be added later to get a proper GRIB1 file."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo -b 24 -a -f grb -t $GRIB1_table copy thetao_med_sea_surface.tmp3.nc thetao_med_sea_surface.tmp1.grb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, set the `variable code` and appropriate `table version` in which the Temperature variable is defined"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"grib_set -s paramId=3080 thetao_med_sea_surface.tmp1.grb -w indicatorOfParameter=80 thetao_med_sea_surface.tmp2.grb"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"grib_set -s table2Version=2 thetao_med_sea_surface.tmp2.grb thetao_med_sea_surface.grb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Get to know your data (freshly created GRIB1 file content)\n",
"---\n",
"In order to check the content of the freshly created GRIB1 file, a few `cdo` command lines are going to be described."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How can I see the value of Water Temperature in my file?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"At first, we're going to extract one gridpoint (using `remapnn` operator) to ease the reading of the output (otherwise we would have 117x69 matrix of gridpoints): "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo remapnn,lon=18/lat=36 thetao_med_sea_surface.grb 1gridpoint.grb && cdo infov 1gridpoint.grb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"FYI: We notice that Water Temperature is well in Kelvin, and not in Celcius Degree as in the initial NetCDF file."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How can I see the timesteps in my file?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": [
"cdo showtimestamp thetao_med_sea_surface.grb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How many timesteps are in my file?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo ntime thetao_med_sea_surface.grb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How to convert temperature units from K to degC?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo addc,-273.15 thetao_med_sea_surface.grb thetao_med_sea_surface_degC.grb"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo sinfon thetao_med_sea_surface_degC.grb"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cdo sinfon thetao_med_sea_surface.grb"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"# To go further: Discovering Copernicus Marine Services <a name=\"discovering-cmems\"></a>\n",
"---\n",
"\n",
"Read at first the [LETS GET STARTED](http://marine.copernicus.eu/services-portfolio/technical-faq/) websection, which goes through:\n",
"\n",
"- Copernicus Marine Product **Standard Output (file format)** is **NetCDF (.nc)**. If you aren’t familiar with this data format, the [netCDF FAQ](http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#What-Is-netCDF) is a good place to start.\n",
" - http://marine.copernicus.eu/faq/format-cmems-products/?idpage=169\n",
"<br>\n",
"<br>\n",
"- Can you provide me with **an overview of available products** in **CMEMS catalogue**?\n",
" - http://marine.copernicus.eu/faq/can-you-provide-me-with-an-overview-of-the-cmems-catalogue-of-products/?idpage=169\n",
" <br>\n",
"<br>\n",
"- How can I access the **Technical** and **Scientific documentation** associated to a product?\n",
" - http://marine.copernicus.eu/faq/how-can-i-access-the-documentation-associated-to-a-product/?idpage=169\n",
" <br>\n",
"<br>\n",
"- How can I get more information regarding **Validation** and **Quality** of **Products**?\n",
" - http://marine.copernicus.eu/faq/how-can-i-get-more-information-regarding-quality-of-products/?idpage=169\n",
" <br>\n",
"<br>\n",
"- How to use the Copernicus Marine online web-GIS tool to **georeference and visualize** a product **BEFORE downloading** the data?\n",
" - http://marine.copernicus.eu/faq/how-to-quickly-visualize-a-product/?idpage=169\n",
" <br>\n",
"<br>\n",
"- How to **download** Copernicus Marine Products?\n",
" - http://marine.copernicus.eu/faq/how-to-download-cmems-products/?idpage=169\n",
" <br>\n",
"<br>\n",
"- What are **the advantages** of each CMEMS **Data Access Options**?\n",
" - http://marine.copernicus.eu/faq/what-are-the-advantages-of-each-cmems-download-mechanism/?idpage=169\n",
" <br>\n",
" <br>\n",
"- Can I **automate downloads** of CMEMS products?\n",
" - http://marine.copernicus.eu/faq/can-i-automate-downloads-of-cmems-products/?idpage=169\n",
" <br>\n",
" <br>\n",
"- What are the **MOTU** and **Python** requirements?\n",
" - http://marine.copernicus.eu/faq/what-are-the-motu-and-python-requirements/?idpage=169 \n",
" <br>\n",
" <br>\n",
"- How **to write** and **run the script** to download CMEMS products through **Subsetter (SUB)** or **Direct download mechanisms (DGF)**?\n",
" - http://marine.copernicus.eu/faq/how-to-write-and-run-the-script-to-download-cmems-products-through-subset-or-direct-download-mechanisms/?idpage=169\n",
" <br>\n",
" <br>\n",
"- Is a **download** request **limited** in **data size**?\n",
" - http://marine.copernicus.eu/faq/is-a-download-limited-in-terms-of-volume/?idpage=169\n",
" <br>\n",
" <br>\n",
"- How to **overcome such limit**?\n",
" - http://forum.marine.copernicus.eu/discussion/202/how-to-download-a-large-amount-of-cmems-data-how-to-perform-a-loop-on-dates-new#Item_1\n",
" - http://marine.copernicus.eu/faq/how-to-download-a-large-amount-of-data-using-subset-or-direct-download-mechanisms/?idpage=169\n",
" <br>\n",
" <br>\n",
"- Can you give a **few examples of command lines** to submit **extraction request** and **dataset subset download**?\n",
" - http://marine.copernicus.eu/faq/can-you-give-a-few-examples-of-command-lines-to-download/?idpage=169\n",
" <br>\n",
" <br>"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.13"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment