Skip to content

Instantly share code, notes, and snippets.

View mdsumner's full-sized avatar

Michael Sumner mdsumner

561 followers · 1.4k following
  • Integrated Digital East Antarctica, Australian Antarctic Division
  • Hobart, Australia
View GitHub Profile
@mdsumner
mdsumner / reticulate_gdal_mdim.md
Created March 6, 2025 00:37 
library(reticulate)
py_require("gdal")
gdal <- import("osgeo.gdal")
gdal$UseExceptions()

f <- "/vsigs/gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3"
Sys.setenv("GS_NO_SIGN_REQUEST" = "YES")
@mdsumner
mdsumner / arraylake_repo_endpoint.md
Created March 4, 2025 21:08 
AWS_REGION='region-is-not-set' al repo create AustralianAntarcticDivision/myrepo
⠴ Creating repo AustralianAntarcticDivision/myrepo...
✓ Creating repo AustralianAntarcticDivision/myrepo...failed
  [31m×[0m error listing objects in object store dispatch failure
  [31m│[0m
  [31m│[0m context:
  [31m│[0m    0: icechunk::repository::create
@mdsumner
mdsumner / gdal_1D_promotion.md
Last active March 4, 2025 11:26

https://rstats.me/@mdsumner/114103904777240036

I've used HDF5 here because NETCDF can't do this on windows. On linux probably best to use "NETCDF:{dsn}:time".

(r <- terra::rast("HDF5:\"/vsicurl/https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198109/oisst-avhrr-v02r01.19810901.nc\"://time"))
class       : SpatRaster 
dimensions  : 1, 1, 1  (nrow, ncol, nlyr)
resolution : 1, 1 (x, y)
@mdsumner
mdsumner / zarr-in-zip.md
Last active February 27, 2025 14:09

convert to Zarr and zip

Interested in using GDAL to convert to zipped Zarr and then open as xarray, or stream through the warper api.

gdalmdimtranslate  /vsicurl/https://projects.pawsey.org.au/idea-10.7289-v5sq8xb5/www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198109/oisst-avhrr-v02r01.19810901.nc \
   abc.zarr -of ZARR

cd abc.zarr
@mdsumner
mdsumner / sp_from_terra.md
Created February 19, 2025 10:14 
cellcentre.offset <- function(x) {
  ex <- terra::ext(x)
  res <- terra::res(x)
  halfcell <- res/2
  cco <- c(ex[1], ex[4]) + halfcell * c(1, -1)
  cco
}
@mdsumner
mdsumner / parallel_rast_extract.md
Last active February 18, 2025 06:03 
service <- file.path(
  "https://s3-us-west-2.amazonaws.com", 
  "mrlc"
)
layers <- paste0(
  service, 
  "/Annual_NLCD_LndCov_", 
  years, 
  "_CU_C1V0.tif"
@mdsumner
mdsumner / spilhaus_proj.md
Last active February 18, 2025 22:18 
dsn <- "/vsicurl/https://projects.pawsey.org.au/idea-gebco-tif/GEBCO_2024.tif"

library(terra)
r <- rast(ext(-1, 1, -1, 1) * 6378137 * 1.9, res = 25000, crs = "+proj=spilhaus")
geb <- project(rast(dsn), r, by_util = TRUE)
@mdsumner
mdsumner / zarr.md
Last active February 17, 2025 23:55

with stars (uses GDAL MULTIDIM)

## note we have to inner-quote the string (I think ZARR driver needs some attention here)
 dsn <- "ZARR:\"/vsicurl/https://mur-sst.s3.us-west-2.amazonaws.com/zarr-v1\""
library(stars)
read_mdim(dsn, proxy = TRUE, bounds = FALSE, variable = "analysed_sst")
stars_proxy object with 1 attribute in 1 file(s):
$analysed_sst
@mdsumner
mdsumner / warp_from_tileserver.md
Created February 17, 2025 21:42 
library(maptiles)
library(sf)
nc = st_read(system.file("shape/nc.shp", package = "sf"))

## let's say we have target raster
rr <-  get_tiles(
    sf::st_transform(nc, "EPSG:3857"),
@mdsumner
mdsumner / gdal_multdim_api.md
Last active February 13, 2025 05:01