mdsumner/polars.md

Created April 20, 2025 22:40

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/mdsumner/13f342ee044b72ecbd071ea4e8166074.js"></script>
Save mdsumner/13f342ee044b72ecbd071ea4e8166074 to your computer and use it in GitHub Desktop.

Raw

## in R before
# p <- c("polars", "h5netcdf", "fsspec", "aiohttp", "requests", "xarray")
# reticulate::py_require(p)

import polars
pp = polars.read_parquet("https://projects.pawsey.org.au/idea-objects/idea-curated-objects.parquet")
d = pp.filter(polars.col("Dataset") == "oisst-avhrr-v02r01")
d = d.with_columns(source = polars.format("https://projects.pawsey.org.au/{}/{}", polars.col("Bucket"), polars.col("Key")))
d0 = d.slice(-1).select("source")
d0.item()

import xarray
xarray.open_dataset(d0.item(), engine = "h5netcdf")