RichardScottOZ

Holy grail

Before diving too deeply into the various friction points when working with archives of earth observation data in xarray, let's look at a more optimal case from the earth systems world. In the notebook here we demonstrate how using zarr's consolidated metadata option to access the dimensional and chunk reference information, a massive dataset's dimensions and variables can be loaded extremely quickly. With this consolidated metadata available to reference chunks on disk, we can leverage xarray's dask integration to use normal xarray operations to lazily load chunks in parallel and perform our calculations using dask's blocked algorithm implementations. Gravy.

Challenges

But the earth observation story is more complicated... Not everything lives in standardized file containers and more importantly our grid coordinate systems are "all over the map" :] Here are some of the current challenges.

Consolida

	from typing import List, Dict, Tuple

	import xarray as xr
	import numpy as np
	import dask.array


	# import pandas as pd
	# def core_dim_locs_from_multiindex(multi_index, coords: Dict[str, pd.Index], new_dim_name, core_dims=None) -> List[
	# Tuple[str, xr.DataArray]]:

	import pymc3 as pm
	import numpy as np
	import arviz as az
	%config InlineBackend.figure_format = 'retina'

	# Data from https://twitter.com/tomstafford/status/1456914037195907079?s=20
	N = np.array([1258, 280]) # total number of caffiene and non-caffiene drinkers
	k = np.array([966, 168]) # total number of those who have favourite mugs

	def cohens_h(p):