mdsumner’s gists

mdsumner / idea-vzarr.md

Last active February 11, 2025 04:16

A small list of global datasets on Pawsey object storage.

'NSIDC_SEAICE_PS_S25km', 'NSIDC_SEAICE_PS_N25km' are the 25km sea ice concentrations between 1978-now.

'SEALEVEL_GLO_PHY_L4' is Copernicus altimetry, global longlat since 1993 (ssh, u/v currents

'oisst-avhrr-v02r01' is OISST 25km global longlat SST since 1982.

Virtual Zarr referencing NetCDF objects on Pawsey storage, stored as "kerchunk Parquet".

mdsumner / goog_storage.md

Last active February 17, 2025 21:17

following on from https://bsky.app/profile/jerry-shannon.bsky.social/post/3lcisatdppk2h

gdalinfo /vsigs/gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3
ERROR 15: No valid GCS credentials found. For authenticated requests, you need to set GS_SECRET_ACCESS_KEY, GS_ACCESS_KEY_ID, 
GS_OAUTH2_REFRESH_TOKEN, GOOGLE_APPLICATION_CREDENTIALS, or other configuration options, or create a /home/ubuntu/.boto file. Consult
https://gdal.org/en/stable/user/virtual_file_systems.html#vsigs-google-cloud-storage-files for more details. For unauthenticated requests
on public resources, set the GS_NO_SIGN_REQUEST configuration option to YES.

mdsumner / sooty_test.md

Created January 29, 2025 02:48

rast("/vsis3/idea-10.7289-v5sq8xb5/www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198109/oisst-avhrr-v02r01.19810901.nc")

mdsumner / split_kml_description.md

Last active January 23, 2025 14:44

## function to split "label: text" into data.frame(label = text)
v1 <- function(x) {  
  x <- gsub(": ", ":", x)  ## first clear that whitespace
  dd <- read.delim(text = x, sep = ":", header = FALSE); 
  setNames(data.frame(dd[[2]]), dd[[1]][1])
}

library(sf)
sf::sf_use_s2(FALSE)

mdsumner / rastplot.md

Created January 23, 2025 00:18

prj <- "+proj=stere +lat_0=-90 +lat_ts=-70 +datum=WGS84"

f <- "~/FastIce_70_2006.nc"
ex <- c(-2700000, -2700000 + 5625 * 1000, -2397000, -2397000 + 4700 * 1000)


library(terra)
r <- rast(f, "Fast_Ice_Time_series")
set.ext(r, ext(ex))

mdsumner / tmap_basemap.md

Last active January 20, 2025 13:46

WIP, trying to get the right resolution from a web tile server

library(tmap)
library(terra)


## reliable world polygons
psrc <- "/vsizip//vsicurl/https://github.com/wmgeolab/geoBoundaries/raw/main/releaseData/CGAZ/geoBoundariesCGAZ_ADM0.zip"
psrcsql <- "SELECT shapeGroup FROM geoBoundariesCGAZ_ADM0 WHERE shapeGroup IN ('AUS')"

mdsumner / fastice_terra.md

Created January 20, 2025 01:45

  prj <- "+proj=stere +lat_0=-90 +lat_ts=-70 +datum=WGS84"

f <- "~/FastIce_70_2006.nc"
ex <- c(-2700000, -2700000 + 5625 * 1000, -2397000, -2397000 + 4700 * 1000)

nc <- ncdf4::nc_open(f)
print(nc) ## tells us metadata like
#> File ~/FastIce_70_2006.nc (NC_FORMAT_NETCDF4):
#>

mdsumner / zarr_r.md

Last active March 5, 2025 14:46

Zarr

zarrs

A real Rust library for Zarr, including virtualization support, Icechunk integration and v3.

GDAL

has its own Zarr v2 and v3 internal implementation, works but

mdsumner / aad_.md

Last active January 16, 2025 05:44

docker run -it --rm pangeo/pangeo-notebook:latest bash

pip install arraylake xarray netcdf4 dask pooch

import xarray as xr
from arraylake import Client

mdsumner / seaice.md

Created January 13, 2025 00:12

library(terra)
dates <- seq(as.Date("2024-01-12"), as.Date("2025-01-11"), by = "1 day")
r <- rast(sds::nsidc_seaice(dates[length(dates)], hemisphere = "north"))
r[r > 2500] <- NA
r <- r/10
plot(r)

Michael Sumner mdsumner

Zarr

GDAL