Skip to content

Instantly share code, notes, and snippets.

@andersy005

andersy005/era5.ipynb

Created June 20, 2019 15:26
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save andersy005/96797c29b9fff3767c960a136b650d8b to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save andersy005/96797c29b9fff3767c960a136b650d8b to your computer and use it in GitHub Desktop.
Download ZIP
RDA ERA5 with intake-esm
Display the source blob
Display the rendered blob
Raw
era5.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# ECMWF ERA5 Data Holdings"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Building ERA5 Collection Catalog"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To build ERA5 collection catalog for ERA5 dataset stored on NCAR's GLADE in `/glade/collections/rda/data/ds630.0`, we follow the same steps that are used when building other intake-esm collection catalogs:\n",
"\n",
"- Define a collection catalog in a YAML or nested dictionary.\n",
"\n",
"- Build the collection catalog by passing the collection definition to `intake.open_esm_metadatastore()`.\n",
"\n",
"- Use the built collection catalog by:\n",
"\n",
" - Querying the collection catalog\n",
" - Loading data into an xarray dataset"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import warnings\n",
"warnings.simplefilter(\"ignore\") # Silence warnings\n",
"import intake"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Getting file listing: RDA-GLADE:ERA5:posix:/glade/collections/rda/data/ds630.0\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "eae37098832c469992a94353693438dd",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"HBox(children=(IntProgress(value=0, description='file listing', max=169144, style=ProgressStyle(description_wi…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"<class 'pandas.core.frame.DataFrame'>\n",
"RangeIndex: 169144 entries, 0 to 169143\n",
"Data columns (total 19 columns):\n",
"direct_access 169144 non-null bool\n",
"file_basename 169144 non-null object\n",
"file_dirname 169144 non-null object\n",
"file_fullpath 169144 non-null object\n",
"forecast_initial_date 48790 non-null object\n",
"forecast_initial_hour 48790 non-null object\n",
"grid 169144 non-null object\n",
"level_type 169130 non-null object\n",
"local_table 169144 non-null object\n",
"product_type 169144 non-null object\n",
"reanalysis_day 120340 non-null object\n",
"reanalysis_month 120340 non-null object\n",
"reanalysis_year 120340 non-null object\n",
"resource 169144 non-null object\n",
"resource_type 169144 non-null object\n",
"stream 169144 non-null object\n",
"variable_id 169144 non-null object\n",
"variable_short_name 169144 non-null object\n",
"variable_type 169130 non-null object\n",
"dtypes: bool(1), object(18)\n",
"memory usage: 23.4+ MB\n",
"None\n",
"Persisting GLADE-RDA-ERA5 at : /glade/u/home/abanihi/.intake_esm/collections/era5/GLADE-RDA-ERA5.era5.csv\n"
]
}
],
"source": [
"col = intake.open_esm_metadatastore(collection_input_definition=\"GLADE-RDA-ERA5\", \n",
" overwrite_existing=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Using the Built Collection\n",
"\n",
"Once the collection catalog is built, we can start querying it and using it to load data into xarray datasets:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"col = intake.open_esm_metadatastore(collection_name='GLADE-RDA-ERA5')"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>direct_access</th>\n",
" <th>file_basename</th>\n",
" <th>file_dirname</th>\n",
" <th>file_fullpath</th>\n",
" <th>forecast_initial_date</th>\n",
" <th>forecast_initial_hour</th>\n",
" <th>grid</th>\n",
" <th>level_type</th>\n",
" <th>local_table</th>\n",
" <th>product_type</th>\n",
" <th>reanalysis_day</th>\n",
" <th>reanalysis_month</th>\n",
" <th>reanalysis_year</th>\n",
" <th>resource</th>\n",
" <th>resource_type</th>\n",
" <th>stream</th>\n",
" <th>variable_id</th>\n",
" <th>variable_short_name</th>\n",
" <th>variable_type</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>True</td>\n",
" <td>e5.oper.fc.sfc.minmax.128_202_mn2t.regn320sc.2...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>2002-02-01</td>\n",
" <td>06:00</td>\n",
" <td>regn320sc</td>\n",
" <td>sfc</td>\n",
" <td>128</td>\n",
" <td>forecast</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>202</td>\n",
" <td>mn2t</td>\n",
" <td>minmax</td>\n",
" </tr>\n",
" <tr>\n",
" <th>1</th>\n",
" <td>True</td>\n",
" <td>e5.oper.fc.sfc.minmax.128_049_10fg.regn320sc.2...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>2002-02-16</td>\n",
" <td>06:00</td>\n",
" <td>regn320sc</td>\n",
" <td>sfc</td>\n",
" <td>128</td>\n",
" <td>forecast</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>49</td>\n",
" <td>10fg</td>\n",
" <td>minmax</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2</th>\n",
" <td>True</td>\n",
" <td>e5.oper.fc.sfc.minmax.128_049_10fg.regn320sc.2...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>2002-02-01</td>\n",
" <td>06:00</td>\n",
" <td>regn320sc</td>\n",
" <td>sfc</td>\n",
" <td>128</td>\n",
" <td>forecast</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>49</td>\n",
" <td>10fg</td>\n",
" <td>minmax</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>True</td>\n",
" <td>e5.oper.fc.sfc.minmax.128_202_mn2t.regn320sc.2...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>2002-02-16</td>\n",
" <td>06:00</td>\n",
" <td>regn320sc</td>\n",
" <td>sfc</td>\n",
" <td>128</td>\n",
" <td>forecast</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>202</td>\n",
" <td>mn2t</td>\n",
" <td>minmax</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4</th>\n",
" <td>True</td>\n",
" <td>e5.oper.fc.sfc.minmax.228_226_mxtpr.regn320sc....</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>2002-02-01</td>\n",
" <td>06:00</td>\n",
" <td>regn320sc</td>\n",
" <td>sfc</td>\n",
" <td>228</td>\n",
" <td>forecast</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>226</td>\n",
" <td>mxtpr</td>\n",
" <td>minmax</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" direct_access file_basename \\\n",
"0 True e5.oper.fc.sfc.minmax.128_202_mn2t.regn320sc.2... \n",
"1 True e5.oper.fc.sfc.minmax.128_049_10fg.regn320sc.2... \n",
"2 True e5.oper.fc.sfc.minmax.128_049_10fg.regn320sc.2... \n",
"3 True e5.oper.fc.sfc.minmax.128_202_mn2t.regn320sc.2... \n",
"4 True e5.oper.fc.sfc.minmax.228_226_mxtpr.regn320sc.... \n",
"\n",
" file_dirname \\\n",
"0 /glade/collections/rda/data/ds630.0/e5.oper.fc... \n",
"1 /glade/collections/rda/data/ds630.0/e5.oper.fc... \n",
"2 /glade/collections/rda/data/ds630.0/e5.oper.fc... \n",
"3 /glade/collections/rda/data/ds630.0/e5.oper.fc... \n",
"4 /glade/collections/rda/data/ds630.0/e5.oper.fc... \n",
"\n",
" file_fullpath forecast_initial_date \\\n",
"0 /glade/collections/rda/data/ds630.0/e5.oper.fc... 2002-02-01 \n",
"1 /glade/collections/rda/data/ds630.0/e5.oper.fc... 2002-02-16 \n",
"2 /glade/collections/rda/data/ds630.0/e5.oper.fc... 2002-02-01 \n",
"3 /glade/collections/rda/data/ds630.0/e5.oper.fc... 2002-02-16 \n",
"4 /glade/collections/rda/data/ds630.0/e5.oper.fc... 2002-02-01 \n",
"\n",
" forecast_initial_hour grid level_type local_table product_type \\\n",
"0 06:00 regn320sc sfc 128 forecast \n",
"1 06:00 regn320sc sfc 128 forecast \n",
"2 06:00 regn320sc sfc 128 forecast \n",
"3 06:00 regn320sc sfc 128 forecast \n",
"4 06:00 regn320sc sfc 228 forecast \n",
"\n",
" reanalysis_day reanalysis_month reanalysis_year \\\n",
"0 NaN NaN NaN \n",
"1 NaN NaN NaN \n",
"2 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"4 NaN NaN NaN \n",
"\n",
" resource resource_type stream \\\n",
"0 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... posix oper \n",
"1 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... posix oper \n",
"2 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... posix oper \n",
"3 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... posix oper \n",
"4 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... posix oper \n",
"\n",
" variable_id variable_short_name variable_type \n",
"0 202 mn2t minmax \n",
"1 49 10fg minmax \n",
"2 49 10fg minmax \n",
"3 202 mn2t minmax \n",
"4 226 mxtpr minmax "
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"col.df.head()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['regn320sc', 'regn320uv'], dtype=object)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Find unique grids\n",
"col.df.grid.unique()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['forecast', 'reanalysis', 'invariant'], dtype=object)"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Find unique product types\n",
"col.df.product_type.unique()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['minmax', 'accumu', 'instan', nan], dtype=object)"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Find variable types\n",
"col.df.variable_type.unique()"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['mn2t', '10fg', 'mxtpr', 'mntpr', 'mx2t', 'ewss', 'tsrc', 'ssrdc',\n",
" 'vimd', 'tp', 'ssrc', 'ssro', 'es', 'ttrc', 'strdc', 'mgws',\n",
" 'lgws', 'smlt', 'lspf', 'ssr', 'strc', 'tsr', 'cp', 'sro', 'csf',\n",
" 'uvb', 'str', 'bld', 'sf', 'strd', 'sshf', 'ttr', 'nsss', 'lsf',\n",
" 'tisr', 'gwd', 'e', 'ssrd', 'pev', 'fdir', 'ro', 'slhf', 'cdir',\n",
" 'lsp', 'viozn', 'vipile', 'viiwd', 'vitoen', 'vige', 'vitoee',\n",
" 'viwvn', 'vithed', 'viman', 'vima', 'vitoed', 'vike', 'vit',\n",
" 'vipie', 'vithen', 'vign', 'vilwe', 'viwve', 'vithee', 'viked',\n",
" 'vimat', 'vilwd', 'viiwn', 'vitoe', 'vioze', 'viozd', 'vimae',\n",
" 'viwvd', 'viec', 'vimad', 'vikee', 'viiwe', 'vithe', 'vigd',\n",
" 'viken', 'vilwn', 'tcwv', 'u10n', 'hcc', 'sp', 'lailv', 'tcrw',\n",
" 'tcw', 'chnk', 'tciw', '10v', 'licd', 'ci', 'stl1', 'swvl1',\n",
" 'cape', 'blh', 'stl4', 'istl3', '2d', 'fal', 'fsr', 'ie', 'flsr',\n",
" 'asn', 'ishf', 'lmld', 'lshf', 'tco3', 'aluvd', 'lblt', 'sstk',\n",
" 'swvl4', '2t', '10u', 'alnid', 'tcc', 'sd', 'aluvp', 'src', 'lict',\n",
" 'lcc', 'lmlt', 'inss', 'istl1', 'rsn', 'v10n', 'swvl2', 'laihv',\n",
" 'swvl3', 'tcsw', 'iews', 'ltlt', 'stl3', 'istl2', 'tsn', 'mcc',\n",
" 'skt', 'msl', 'istl4', 'alnip', '100u', 'stl2', 'tclw', '100v',\n",
" 'crwc', 'o3', 'd', 'v', 'z', 't', 'ciwc', 'w', 'r', 'u', 'cswc',\n",
" 'clwc', 'q', 'pv', 'vo', 'cc', 'isor', 'dl', 'tvl', 'slor',\n",
" 'sdfor', 'tvh', 'cl', 'slt', 'cvh', 'cvl', 'sdor', 'lsm', 'anor',\n",
" 'cin', 'csfr', 'lssfr', 'ilspf', 'cbh', 'i10fg', 'crr', 'lsrr',\n",
" 'tplb', 'dndza', 'dndzn', 'deq01', 'zust', 'dctb', 'tplt'],\n",
" dtype=object)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Find available variables (parameters)\n",
"col.df.variable_short_name.unique()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Querying the collection catalog"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Retrive the following `forecast` variables at specific times:\n",
"\n",
" - `mn2t`: Maximum temperature at 2 metres since previous post-processing\n",
" - `mx2t`: Minimum temperature at 2 metres since previous post-processing"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"cat = col.search(variable_short_name=['mn2t', 'mx2t'], \n",
" forecast_initial_date=[\"2002-02-01\", \"2002-02-16\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Retrive the following `reanalysis` variables at specific times:\n",
"\n",
" - `pv`: Potential vorticity and \n",
" - `crwc`: Specific rain water content variables \n",
" - `2t`: 2 metre temperature\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"cat2 = col.search(variable_short_name=['pv', 'crwc', '2t'], reanalysis_day=[1, 2], \n",
" reanalysis_year=2002, reanalysis_month=2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"<div class=\"alert alert-info\">\n",
"\n",
"**Note:** \n",
" \n",
"Full information on all ECMWF parameters/variables is available from the [ECMWF parameter database](http://apps.ecmwf.int/codes/grib/param-db).\n",
"\n",
"\n",
"\n",
"</div>\n",
" \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `search()` method returns a catalog entry. This catalog entry has a dataframe `query_results` attribute that contains the search results:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>direct_access</th>\n",
" <th>file_basename</th>\n",
" <th>file_dirname</th>\n",
" <th>file_fullpath</th>\n",
" <th>forecast_initial_date</th>\n",
" <th>forecast_initial_hour</th>\n",
" <th>grid</th>\n",
" <th>level_type</th>\n",
" <th>local_table</th>\n",
" <th>product_type</th>\n",
" <th>reanalysis_day</th>\n",
" <th>reanalysis_month</th>\n",
" <th>reanalysis_year</th>\n",
" <th>resource</th>\n",
" <th>resource_type</th>\n",
" <th>stream</th>\n",
" <th>variable_id</th>\n",
" <th>variable_short_name</th>\n",
" <th>variable_type</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>7</th>\n",
" <td>True</td>\n",
" <td>e5.oper.fc.sfc.minmax.128_201_mx2t.regn320sc.2...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>2002-02-01</td>\n",
" <td>06:00</td>\n",
" <td>regn320sc</td>\n",
" <td>sfc</td>\n",
" <td>128</td>\n",
" <td>forecast</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>201</td>\n",
" <td>mx2t</td>\n",
" <td>minmax</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8</th>\n",
" <td>True</td>\n",
" <td>e5.oper.fc.sfc.minmax.128_201_mx2t.regn320sc.2...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>2002-02-16</td>\n",
" <td>06:00</td>\n",
" <td>regn320sc</td>\n",
" <td>sfc</td>\n",
" <td>128</td>\n",
" <td>forecast</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>201</td>\n",
" <td>mx2t</td>\n",
" <td>minmax</td>\n",
" </tr>\n",
" <tr>\n",
" <th>0</th>\n",
" <td>True</td>\n",
" <td>e5.oper.fc.sfc.minmax.128_202_mn2t.regn320sc.2...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>2002-02-01</td>\n",
" <td>06:00</td>\n",
" <td>regn320sc</td>\n",
" <td>sfc</td>\n",
" <td>128</td>\n",
" <td>forecast</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>202</td>\n",
" <td>mn2t</td>\n",
" <td>minmax</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3</th>\n",
" <td>True</td>\n",
" <td>e5.oper.fc.sfc.minmax.128_202_mn2t.regn320sc.2...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.fc...</td>\n",
" <td>2002-02-16</td>\n",
" <td>06:00</td>\n",
" <td>regn320sc</td>\n",
" <td>sfc</td>\n",
" <td>128</td>\n",
" <td>forecast</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>202</td>\n",
" <td>mn2t</td>\n",
" <td>minmax</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" direct_access file_basename \\\n",
"7 True e5.oper.fc.sfc.minmax.128_201_mx2t.regn320sc.2... \n",
"8 True e5.oper.fc.sfc.minmax.128_201_mx2t.regn320sc.2... \n",
"0 True e5.oper.fc.sfc.minmax.128_202_mn2t.regn320sc.2... \n",
"3 True e5.oper.fc.sfc.minmax.128_202_mn2t.regn320sc.2... \n",
"\n",
" file_dirname \\\n",
"7 /glade/collections/rda/data/ds630.0/e5.oper.fc... \n",
"8 /glade/collections/rda/data/ds630.0/e5.oper.fc... \n",
"0 /glade/collections/rda/data/ds630.0/e5.oper.fc... \n",
"3 /glade/collections/rda/data/ds630.0/e5.oper.fc... \n",
"\n",
" file_fullpath forecast_initial_date \\\n",
"7 /glade/collections/rda/data/ds630.0/e5.oper.fc... 2002-02-01 \n",
"8 /glade/collections/rda/data/ds630.0/e5.oper.fc... 2002-02-16 \n",
"0 /glade/collections/rda/data/ds630.0/e5.oper.fc... 2002-02-01 \n",
"3 /glade/collections/rda/data/ds630.0/e5.oper.fc... 2002-02-16 \n",
"\n",
" forecast_initial_hour grid level_type local_table product_type \\\n",
"7 06:00 regn320sc sfc 128 forecast \n",
"8 06:00 regn320sc sfc 128 forecast \n",
"0 06:00 regn320sc sfc 128 forecast \n",
"3 06:00 regn320sc sfc 128 forecast \n",
"\n",
" reanalysis_day reanalysis_month reanalysis_year \\\n",
"7 NaN NaN NaN \n",
"8 NaN NaN NaN \n",
"0 NaN NaN NaN \n",
"3 NaN NaN NaN \n",
"\n",
" resource resource_type stream \\\n",
"7 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... posix oper \n",
"8 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... posix oper \n",
"0 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... posix oper \n",
"3 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... posix oper \n",
"\n",
" variable_id variable_short_name variable_type \n",
"7 201 mx2t minmax \n",
"8 201 mx2t minmax \n",
"0 202 mn2t minmax \n",
"3 202 mn2t minmax "
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat.query_results.head()"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>direct_access</th>\n",
" <th>file_basename</th>\n",
" <th>file_dirname</th>\n",
" <th>file_fullpath</th>\n",
" <th>forecast_initial_date</th>\n",
" <th>forecast_initial_hour</th>\n",
" <th>grid</th>\n",
" <th>level_type</th>\n",
" <th>local_table</th>\n",
" <th>product_type</th>\n",
" <th>reanalysis_day</th>\n",
" <th>reanalysis_month</th>\n",
" <th>reanalysis_year</th>\n",
" <th>resource</th>\n",
" <th>resource_type</th>\n",
" <th>stream</th>\n",
" <th>variable_id</th>\n",
" <th>variable_short_name</th>\n",
" <th>variable_type</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>38978</th>\n",
" <td>True</td>\n",
" <td>e5.oper.an.pl.128_060_pv.regn320sc.2002020100_...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.an...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.an...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>regn320sc</td>\n",
" <td>pl</td>\n",
" <td>128</td>\n",
" <td>reanalysis</td>\n",
" <td>1.0</td>\n",
" <td>2.0</td>\n",
" <td>2002.0</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>60</td>\n",
" <td>pv</td>\n",
" <td>instan</td>\n",
" </tr>\n",
" <tr>\n",
" <th>38595</th>\n",
" <td>True</td>\n",
" <td>e5.oper.an.pl.128_060_pv.regn320sc.2002020200_...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.an...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.an...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>regn320sc</td>\n",
" <td>pl</td>\n",
" <td>128</td>\n",
" <td>reanalysis</td>\n",
" <td>2.0</td>\n",
" <td>2.0</td>\n",
" <td>2002.0</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>60</td>\n",
" <td>pv</td>\n",
" <td>instan</td>\n",
" </tr>\n",
" <tr>\n",
" <th>38840</th>\n",
" <td>True</td>\n",
" <td>e5.oper.an.pl.128_075_crwc.regn320sc.200202010...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.an...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.an...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>regn320sc</td>\n",
" <td>pl</td>\n",
" <td>128</td>\n",
" <td>reanalysis</td>\n",
" <td>1.0</td>\n",
" <td>2.0</td>\n",
" <td>2002.0</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>75</td>\n",
" <td>crwc</td>\n",
" <td>instan</td>\n",
" </tr>\n",
" <tr>\n",
" <th>38654</th>\n",
" <td>True</td>\n",
" <td>e5.oper.an.pl.128_075_crwc.regn320sc.200202020...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.an...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.an...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>regn320sc</td>\n",
" <td>pl</td>\n",
" <td>128</td>\n",
" <td>reanalysis</td>\n",
" <td>2.0</td>\n",
" <td>2.0</td>\n",
" <td>2002.0</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>75</td>\n",
" <td>crwc</td>\n",
" <td>instan</td>\n",
" </tr>\n",
" <tr>\n",
" <th>25452</th>\n",
" <td>True</td>\n",
" <td>e5.oper.an.sfc.128_167_2t.regn320sc.2002020100...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.an...</td>\n",
" <td>/glade/collections/rda/data/ds630.0/e5.oper.an...</td>\n",
" <td>NaN</td>\n",
" <td>NaN</td>\n",
" <td>regn320sc</td>\n",
" <td>sfc</td>\n",
" <td>128</td>\n",
" <td>reanalysis</td>\n",
" <td>1.0</td>\n",
" <td>2.0</td>\n",
" <td>2002.0</td>\n",
" <td>RDA-GLADE:ERA5:posix:/glade/collections/rda/da...</td>\n",
" <td>posix</td>\n",
" <td>oper</td>\n",
" <td>167</td>\n",
" <td>2t</td>\n",
" <td>instan</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" direct_access file_basename \\\n",
"38978 True e5.oper.an.pl.128_060_pv.regn320sc.2002020100_... \n",
"38595 True e5.oper.an.pl.128_060_pv.regn320sc.2002020200_... \n",
"38840 True e5.oper.an.pl.128_075_crwc.regn320sc.200202010... \n",
"38654 True e5.oper.an.pl.128_075_crwc.regn320sc.200202020... \n",
"25452 True e5.oper.an.sfc.128_167_2t.regn320sc.2002020100... \n",
"\n",
" file_dirname \\\n",
"38978 /glade/collections/rda/data/ds630.0/e5.oper.an... \n",
"38595 /glade/collections/rda/data/ds630.0/e5.oper.an... \n",
"38840 /glade/collections/rda/data/ds630.0/e5.oper.an... \n",
"38654 /glade/collections/rda/data/ds630.0/e5.oper.an... \n",
"25452 /glade/collections/rda/data/ds630.0/e5.oper.an... \n",
"\n",
" file_fullpath \\\n",
"38978 /glade/collections/rda/data/ds630.0/e5.oper.an... \n",
"38595 /glade/collections/rda/data/ds630.0/e5.oper.an... \n",
"38840 /glade/collections/rda/data/ds630.0/e5.oper.an... \n",
"38654 /glade/collections/rda/data/ds630.0/e5.oper.an... \n",
"25452 /glade/collections/rda/data/ds630.0/e5.oper.an... \n",
"\n",
" forecast_initial_date forecast_initial_hour grid level_type \\\n",
"38978 NaN NaN regn320sc pl \n",
"38595 NaN NaN regn320sc pl \n",
"38840 NaN NaN regn320sc pl \n",
"38654 NaN NaN regn320sc pl \n",
"25452 NaN NaN regn320sc sfc \n",
"\n",
" local_table product_type reanalysis_day reanalysis_month \\\n",
"38978 128 reanalysis 1.0 2.0 \n",
"38595 128 reanalysis 2.0 2.0 \n",
"38840 128 reanalysis 1.0 2.0 \n",
"38654 128 reanalysis 2.0 2.0 \n",
"25452 128 reanalysis 1.0 2.0 \n",
"\n",
" reanalysis_year resource \\\n",
"38978 2002.0 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... \n",
"38595 2002.0 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... \n",
"38840 2002.0 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... \n",
"38654 2002.0 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... \n",
"25452 2002.0 RDA-GLADE:ERA5:posix:/glade/collections/rda/da... \n",
"\n",
" resource_type stream variable_id variable_short_name variable_type \n",
"38978 posix oper 60 pv instan \n",
"38595 posix oper 60 pv instan \n",
"38840 posix oper 75 crwc instan \n",
"38654 posix oper 75 crwc instan \n",
"25452 posix oper 167 2t instan "
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cat2.query_results.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Loading data into an xarray dataset"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "01b5e826555845499139911bc11a7a8e",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"HBox(children=(IntProgress(value=0, description='product', max=1, style=ProgressStyle(description_width='initi…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "98ca64cd25314530b06cfca518fc56aa",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"HBox(children=(IntProgress(value=0, description='variable', max=2, style=ProgressStyle(description_width='init…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"CPU times: user 1.77 s, sys: 178 ms, total: 1.95 s\n",
"Wall time: 1.94 s\n"
]
}
],
"source": [
"%%time \n",
"\n",
"ds_forecast = cat.to_xarray(chunks={'forecast_initial_time': 10}, \n",
" time_coord_name='forecast_initial_time')"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict_keys(['forecast'])"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds_forecast.keys()"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<xarray.Dataset>\n",
"Dimensions: (forecast_hour: 19, forecast_initial_time: 56, latitude: 640, longitude: 1280)\n",
"Coordinates:\n",
" * forecast_hour (forecast_hour) int32 0 1 2 3 4 5 ... 14 15 16 17 18\n",
" * latitude (latitude) float64 89.78 89.51 ... -89.51 -89.78\n",
" * longitude (longitude) float64 0.0 0.2812 0.5625 ... 359.4 359.7\n",
" * forecast_initial_time (forecast_initial_time) datetime64[ns] 2002-02-01T06:00:00 ... 2002-02-28T18:00:00\n",
" utc_date (forecast_initial_time) int32 dask.array<shape=(56,), chunksize=(10,)>\n",
" ANOR (latitude, longitude) float32 0.78539824 ... -0.060503244\n",
" CL (latitude, longitude) float32 0.0 0.0 0.0 ... 0.0 0.0\n",
" CVH (latitude, longitude) float32 0.0 0.0 0.0 ... 0.0 0.0\n",
" CVL (latitude, longitude) float32 0.0 0.0 0.0 ... 0.0 0.0\n",
" DL (latitude, longitude) float32 4192.6973 ... 25.0\n",
" ISOR (latitude, longitude) float32 0.0 0.0 ... 0.075000286\n",
" LSM (latitude, longitude) float32 0.0 0.0 0.0 ... 1.0 1.0\n",
" SDFOR (latitude, longitude) float32 0.0 0.0 ... 0.328125\n",
" SDOR (latitude, longitude) float32 0.0 0.0 ... 3.3833008\n",
" SLOR (latitude, longitude) float32 9.9999976e-05 ... 0.0020719452\n",
" SLT (latitude, longitude) float32 0.0 0.0 0.0 ... 1.0 1.0\n",
" TVH (latitude, longitude) float32 0.0 0.0 0.0 ... 0.0 0.0\n",
" TVL (latitude, longitude) float32 0.0 0.0 0.0 ... 0.0 0.0\n",
" Z (latitude, longitude) float32 5.503662 ... 27189.738\n",
"Data variables:\n",
" MN2T (forecast_initial_time, forecast_hour, latitude, longitude) float32 dask.array<shape=(56, 19, 640, 1280), chunksize=(10, 19, 640, 1280)>\n",
" MX2T (forecast_initial_time, forecast_hour, latitude, longitude) float32 dask.array<shape=(56, 19, 640, 1280), chunksize=(10, 19, 640, 1280)>\n",
"Attributes:\n",
" Conventions: CF-1.6\n",
" CONVERSION_DATE: Fri Nov 10 14:31:30 MST 2017\n",
" NETCDF_COMPRESSION: NCO: Precision-preserving compression to netCDF4/HDF...\n",
" history: 2019-06-20 09:23:03.984390 xarray.open_dataset('/gla...\n",
" DATA_SOURCE: ECMWF: ERA5 GRIB 1 spherical harmonics (T639) and re...\n",
" NETCDF_CONVERSION: CISL RDA: Conversion from interpolated/synthesized G..."
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds = ds_forecast['forecast']\n",
"ds"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "572787fa9b3f426cadc0670133778d05",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"HBox(children=(IntProgress(value=0, description='product', max=1, style=ProgressStyle(description_width='initi…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c09dbd9b2c54441e8e789ca3ec0f073d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"HBox(children=(IntProgress(value=0, description='variable', max=3, style=ProgressStyle(description_width='init…"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"CPU times: user 1.79 s, sys: 168 ms, total: 1.96 s\n",
"Wall time: 2.3 s\n"
]
}
],
"source": [
"%time ds_reanalysis = cat2.to_xarray(chunks={'time': 20})"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict_keys(['reanalysis'])"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds_reanalysis.keys()"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<xarray.Dataset>\n",
"Dimensions: (latitude: 640, level: 37, longitude: 1280, time: 672)\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 2002-02-01 ... 2002-02-28T23:00:00\n",
" * latitude (latitude) float64 89.78 89.51 89.23 ... -89.23 -89.51 -89.78\n",
" * longitude (longitude) float64 0.0 0.2812 0.5625 ... 359.2 359.4 359.7\n",
" utc_date (time) int32 dask.array<shape=(672,), chunksize=(20,)>\n",
" * level (level) float64 1.0 2.0 3.0 5.0 7.0 ... 925.0 950.0 975.0 1e+03\n",
" ANOR (latitude, longitude) float32 0.78539824 ... -0.060503244\n",
" CL (latitude, longitude) float32 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0\n",
" CVH (latitude, longitude) float32 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0\n",
" CVL (latitude, longitude) float32 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0\n",
" DL (latitude, longitude) float32 4192.6973 4192.1475 ... 25.0 25.0\n",
" ISOR (latitude, longitude) float32 0.0 0.0 ... 0.076232135 0.075000286\n",
" LSM (latitude, longitude) float32 0.0 0.0 0.0 0.0 ... 1.0 1.0 1.0 1.0\n",
" SDFOR (latitude, longitude) float32 0.0 0.0 0.0 ... 0.32861328 0.328125\n",
" SDOR (latitude, longitude) float32 0.0 0.0 0.0 ... 3.3911133 3.3833008\n",
" SLOR (latitude, longitude) float32 9.9999976e-05 ... 0.0020719452\n",
" SLT (latitude, longitude) float32 0.0 0.0 0.0 0.0 ... 1.0 1.0 1.0 1.0\n",
" TVH (latitude, longitude) float32 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0\n",
" TVL (latitude, longitude) float32 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0\n",
" Z (latitude, longitude) float32 5.503662 5.519287 ... 27189.738\n",
"Data variables:\n",
" VAR_2T (time, latitude, longitude) float32 dask.array<shape=(672, 640, 1280), chunksize=(20, 640, 1280)>\n",
" CRWC (time, level, latitude, longitude) float32 dask.array<shape=(672, 37, 640, 1280), chunksize=(20, 37, 640, 1280)>\n",
" PV (time, level, latitude, longitude) float32 dask.array<shape=(672, 37, 640, 1280), chunksize=(20, 37, 640, 1280)>\n",
"Attributes:\n",
" Conventions: CF-1.6\n",
" NETCDF_COMPRESSION: NCO: Precision-preserving compression to netCDF4/HDF...\n",
" history: 2019-06-20 09:23:32.555558 xarray.open_dataset('/gla...\n",
" DATA_SOURCE: ECMWF: ERA5 GRIB 1 spherical harmonics (T639) and re...\n",
" NETCDF_CONVERSION: CISL RDA: Conversion from interpolated/synthesized G..."
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"ds_reanalysis['reanalysis']"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment