mdsumner’s gists

mdsumner / tinydemo_aad_dataset.md

Last active November 15, 2024 05:49

These are two datasets, the Reynolds .25deg OISST and the NSIDC 25km southern polar sea ice concentration, up to date for those collections and updated daily as new data are published.

These are synchronized by {bowerbird} to disk and object storage and delivered to R users via {raadtools} on openstack VMs, or at HQ.

This VirtualiZarr-ing of these datasets is incremental progress towards opening up raadtools to the public and cross language usage.

import xarray

## public bucket on Pawsey endpoint

mdsumner / ant_antarctica.md

Last active November 13, 2024 11:52

R code for copying this cool toot: https://en.osm.town/@koriander/113459280926992782

src <- "/vsizip//vsicurl/https://github.com/wmgeolab/geoBoundaries/raw/main/releaseData/CGAZ/geoBoundariesCGAZ_ADM0.zip"
sql <- "SELECT shapeGroup FROM geoBoundariesCGAZ_ADM0 WHERE shapeGroup IN ('ATA')"
library(terra)
laea <- project(vect(src), "+proj=laea +lat_0=90")

ant &lt;- project(vect(src, query = sql), "+proj=laea +lat_0=-90")

mdsumner / bluelink.md

Created November 13, 2024 00:20

      ## see here: https://thredds.nci.org.au/thredds/catalog/gb6/BRAN/BRAN2020/catalog.html

## under daily, the HTTPServer netcdf looks like this: 

mldsrc <- "https://thredds.nci.org.au/thredds/fileServer/gb6/BRAN/BRAN2020/daily/ocean_mld_2023_12.nc"

library(terra)
#> terra 1.7.78
## with gdal we can do

mdsumner / vtiff.md

Last active November 10, 2024 03:50

def open_virtual(filepath, creds): ds = open_virtual_dataset(filepath, indexes = {}, loadable_variables=['x', 'y', 'time', 'crs'], decode_times = True, reader_options={'storage_options': creds})

zarr-developers/VirtualiZarr#291

mdsumner / lyptych.md

Created November 8, 2024 01:17

    library(terra)

options(parallelly.fork.enable = TRUE, future.rng.onMisuse = "ignore")
library(furrr)

topo <- project(rast(sds::gebco()), rast(), by_util = TRUE)
plot(topo)

mdsumner / spartacus.md

Created November 7, 2024 13:28

https://kafeneio.social/@IAmSpartacus/113440758931805059

mdsumner / urlparquet_rpy.md

Last active November 11, 2024 10:04

Read parquet from S3 in python:

import pyarrow.parquet as pq
from pyarrow import fs

aws_credentials = {"endpoint_url": "https://projects.pawsey.org.au",  "anon": True}

s3 = fs.S3FileSystem(endpoint_override = "projects.pawsey.org.au")

mdsumner / a.md

Created November 4, 2024 23:00

abc

mdsumner / vzarr_test.md

Last active November 4, 2024 20:05

Netcdf files in public bucket idea-10.7289-v5sq8xb5/ on Pawsey endpoint, but we don't need netcdf with the VirtualiZarr references in Parquet:

import xarray

## public bucket on Pawsey endpoint
so = {"endpoint_url": "https://projects.pawsey.org.au",  "anon": True}


import xarray

mdsumner / healpix.md

Created October 28, 2024 05:18

terra for healpix of GEBCO

s <- "/vsicurl/https://gebco2023.s3.valeria.science/gebco_2023_land_cog.tif"
library(terra)
A <- 20037508.342789244
target <- rast(ext(c(-1, 1, -.5, .5) * A), res = 25000, crs = "+proj=healpix")

r <- project(rast(s), target, by_util = TRUE)
plot(r)

Michael Sumner mdsumner