Michael Sumner mdsumner

small example using a docker image from https://github.com/hypertidy/gdal-r-ci, I'm using dev because the mdim print output is human readable (by default, json by option)

docker pull ghcr.io/hypertidy/gdal-system:dev
docker run --rm -ti ghcr.io/hypertidy/gdal-system:dev

#── ghcr.io/hypertidy/gdal-system ──
#GDAL 3.14.0dev   PROJ 9.8.1   GEOS 3.15.0beta1
#Py   3.12.3   numpy 2.5.1

xy <- do.call(cbind, maps::map(plot = F)[1:2])

mm <- matrix(c(8L, -95L, -50L, 59L, -47L, -62L), ncol = 2)
mm <- rbind(mm, mm[1, ])
n <- nrow(mm)
x <- 1:n
idx <- seq(1, n, length.out = 200 )
par(mfrow = c(2, 1))
maps::map()

redirect_storage panics (PanicException, not IcechunkError) resolving an http+icechunk:// Location

Following the storage guide's Redirect Storage section: a 302 whose Location uses the documented http+icechunk:// scheme causes a Rust panic that escapes to Python as pyo3_runtime.PanicException rather than an IcechunkError.

Reprex:

import http.server, threading, icechunk as ic

List all of OISST netcdf files from object storage and derive date from path
Remove duplicates that are 'preliminary' files replaced by final, and sort by date
Stack into MDIM VRT with xml and templating from the first few files

### 1. 
Sys.setenv("AWS_NO_SIGN_REQUEST" = "YES")
root <- "/vsis3/noaa-cdr-sea-surface-temp-optimum-interpolation-pds/data/v2.1/avhrr"

#run --rm -ti -v $HOME/Git/gdalxarray:/gdalxarray  ghcr.io/hypertidy/gdal-r-python:dev


import os
os.environ["AWS_NO_SIGN_REQUEST"] = "YES"
os.environ["AWS_REGION"] = "us-west-2"

# 1. confirm register() is active and chunk manager is happy
from dask_array.xarray import register; register()

docker run --rm  ghcr.io/hypertidy/gdal-system:dev sh -c "gdal mdim info /vsis3/dynamical-noaa-gefs/noaa-gefs-analysis/v0.1.2.icechunk --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-west-2 "

Driver: Icechunk

Dimensions:

NOAA GEFS analysis on dynamical.org in earthmover marketplace: https://app.earthmover.io/marketplace/6970566255e09e23d5bcbbc0?tab=dataset

Open an icechunk store in gdalxarray, using the icechunk GDAL driver

# docker run --rm -ti ghcr.io/hypertidy/gdal-r-python:dev 
## uv pip install gdalxarray (if docker image not updated yet)

import xarray
import os

vrt = gdalxarray.warp("/vsicurl/https://projects.pawsey.org.au/idea-gebco-tif/GEBCO_2024.tif", 
     cutlineDSName =  "/vsicurl/https://github.com/mdsumner/geoboundaries/releases/download/latest/geoBoundariesCGAZ_ADM0.parquet",      
     cutlineSQL = "SELECT shapeGroup FROM geoBoundariesCGAZ_ADM0 WHERE shapeGroup IN ('AUS')",  
     cropToCutline = True, resolution = 0.1)
ds = xarray.open_dataset(vrt, engine = "gdalxarray", multidim = False)
ds
<xarray.Dataset> Size: 858kB
Dimensions: (band: 1, y: 447, x: 947)

## list the structure elements (Icechunk) gdal vsi ls  "/vsis3/earthmover-icechunk-era5/icechunkV2" --config AWS_NO_SIGN_REQUEST YES --config AWS_REGION us-east-1
repo
chunks
manifests
overwritten
snapshots
transactions

	```
	variables:
	float stereographic ;
	stereographic:missing_value = -1.e+34f ;
	stereographic:_FillValue = -1.e+34f ;
	stereographic:long_name = "Antarctic Polar Stereographic" ;
	stereographic:grid_mapping = "polar_stereographic" ;
	stereographic:standard_name = "polar_stereographic" ;
	stereographic:authority = "EPSG:3031" ;
	stereographic:latitude_of_origin = -71.f ;