mdsumner

These are two datasets, the Reynolds .25deg OISST and the NSIDC 25km southern polar sea ice concentration, up to date for those collections and updated daily as new data are published.

These are synchronized by {bowerbird} to disk and object storage and delivered to R users via {raadtools} on openstack VMs, or at HQ.

This VirtualiZarr-ing of these datasets is incremental progress towards opening up raadtools to the public and cross language usage.

import xarray

## public bucket on Pawsey endpoint

mdsumner

R code for copying this cool toot: https://en.osm.town/@koriander/113459280926992782

src <- "/vsizip//vsicurl/https://github.com/wmgeolab/geoBoundaries/raw/main/releaseData/CGAZ/geoBoundariesCGAZ_ADM0.zip"
sql <- "SELECT shapeGroup FROM geoBoundariesCGAZ_ADM0 WHERE shapeGroup IN ('ATA')"
library(terra)
laea <- project(vect(src), "+proj=laea +lat_0=90")

ant &lt;- project(vect(src, query = sql), "+proj=laea +lat_0=-90")

mdsumner

  ## see here: https://thredds.nci.org.au/thredds/catalog/gb6/BRAN/BRAN2020/catalog.html

## under daily, the HTTPServer netcdf looks like this: 

mldsrc <- "https://thredds.nci.org.au/thredds/fileServer/gb6/BRAN/BRAN2020/daily/ocean_mld_2023_12.nc"

library(terra)
#> terra 1.7.78
## with gdal we can do 

mdsumner

def open_virtual(filepath, creds): ds = open_virtual_dataset(filepath, indexes = {}, loadable_variables=['x', 'y', 'time', 'crs'], decode_times = True, reader_options={'storage_options': creds})

zarr-developers/VirtualiZarr#291

mdsumner

library(terra)

options(parallelly.fork.enable = TRUE, future.rng.onMisuse = "ignore")
library(furrr)



topo <- project(rast(sds::gebco()), rast(), by_util = TRUE)
plot(topo)

mdsumner

https://kafeneio.social/@IAmSpartacus/113440758931805059

mdsumner

Read parquet from S3 in python:

import pyarrow.parquet as pq
from pyarrow import fs

aws_credentials = {"endpoint_url": "https://projects.pawsey.org.au",  "anon": True}

s3 = fs.S3FileSystem(endpoint_override = "projects.pawsey.org.au")

mdsumner

abc

mdsumner

Netcdf files in public bucket idea-10.7289-v5sq8xb5/ on Pawsey endpoint, but we don't need netcdf with the VirtualiZarr references in Parquet:

import xarray

## public bucket on Pawsey endpoint
so = {"endpoint_url": "https://projects.pawsey.org.au",  "anon": True}


import xarray

mdsumner

terra for healpix of GEBCO

s <- "/vsicurl/https://gebco2023.s3.valeria.science/gebco_2023_land_cog.tif"
library(terra)
A <- 20037508.342789244
target <- rast(ext(c(-1, 1, -.5, .5) * A), res = 25000, crs = "+proj=healpix")

r <- project(rast(s), target, by_util = TRUE)
plot(r)