RichardScottOZ

Holy grail

Before diving too deeply into the various friction points when working with archives of earth observation data in xarray, let's look at a more optimal case from the earth systems world. In the notebook here we demonstrate how using zarr's consolidated metadata option to access the dimensional and chunk reference information, a massive dataset's dimensions and variables can be loaded extremely quickly. With this consolidated metadata available to reference chunks on disk, we can leverage xarray's dask integration to use normal xarray operations to lazily load chunks in parallel and perform our calculations using dask's blocked algorithm implementations. Gravy.

Challenges

But the earth observation story is more complicated... Not everything lives in standardized file containers and more importantly our grid coordinate systems are "all over the map" :] Here are some of the current challenges.

Consolida

Packing Conda Environments

You must be using conda for this approach. You will need conda installed on the Source machine and the Target machine. The Source machine must have an internet connection, the Target does not. The OS in both environments must match; no going from macOS to Win10 for example.

1. (Source) Install conda-pack in your base python environment.

conda install -c conda-forge conda-pack

Installing GDAL 3.2.1 on Amazon Linux 2

As of this day, this is probably the only and fastest way of installing it.

Package requirements

Based from the GDAL and PROJ build requirements, here is the full list of required packages to install:

	from typing import List, Dict, Tuple

	import xarray as xr
	import numpy as np
	import dask.array


	# import pandas as pd
	# def core_dim_locs_from_multiindex(multi_index, coords: Dict[str, pd.Index], new_dim_name, core_dims=None) -> List[
	# Tuple[str, xr.DataArray]]:

	import s3fs
	import zipfile
	import tempfile
	import numpy as np
	from tensorflow import keras
	from pathlib import Path
	import logging


	AWS_ACCESS_KEY="aws_access_key"