Skip to content

Instantly share code, notes, and snippets.

@VeckoTheGecko
Last active July 15, 2026 04:51
Show Gist options
  • Select an option

  • Save VeckoTheGecko/e189504a6d77359072722178cd8c10a2 to your computer and use it in GitHub Desktop.

Select an option

Save VeckoTheGecko/e189504a6d77359072722178cd8c10a2 to your computer and use it in GitHub Desktop.
Scipy 2026 Pangeo Tools Plenary

Hey everyone! Pangeo is a community promoting scalable geoscience. We are scientists, software developers, maintainers, and more. We are from all over the world, from big teams and small.

The data we mainly work with are large array datasets on the order of terrabytes (think satellite data, drone data, climate model output). We care a lot about how to efficiently store, update, visualise and analyse these large datasets. So we care a lot about working in the cloud: using Analysis Ready Cloud Optimized (ARCO) data and developing cloud workflows.

And of course there are also some other challenges sprinkled in there since we have historical data, as well as various interesting data structures and formats.

Here what I'll give is a quick snapshot of what people have been up to in this space over the last year. At the QR code there is this full presentation complete with lots of links in case you want to dive into anything in particular here.


STAC is now an OGC Community Standard (Oct 2025 - blog post). This focuses the community around the STAC specification making it easier to specify, and browse dataset catalogues. The European Space Agency has released the EOPF Sentinel Zarr Explorer (Feb 2026 - blog post, portal) demonstrating the power of combining STAC and Zarr.

Icechunk v2 has been released (April 2026 - blog post) providing better consistency, performance and reliability. Enabling new features such as fine-grained data sharing (April 2026 - blog post). Icechunk has also been seeing increased adoption, such as by the US National Weather Service (June 2026 - blog post), and has also been used to virtualise and provide low latency access to large datasets such as NOAA's GOES-16 (June 2026 - blog post) and ERA5 (June 2026 - blog post).

We also had the release of the obstore package (Aug 2025 - blog post, YouTube video), helping streamline and reduce overhead when fetching files from object storage.

We also had the release of zarr-layer (Apr 2026 - blog post) which enables easier browser visualisation of Zarr datasets at multiple resolutions with GPU rendering. Importantly, without having to duplicate or "tile" your data.

And for those still needing tiling, there was the release of xpublish-tiles (Oct 2025 - YouTube) which is a plugin for the xpublish package, taking Xarray datasets and generating mapping tiles for various mesh geometries to be served via a fast-API server to users.

In Project Pythia - the educational arm of Pangeo - there have also been some updates. Project Pythia is now a NumFOCUS Affiliated Project (blog post). Their annual summer hackathon in Boulder Colorado had 50 participants all in person (blog post), also resulting in a refreshed Pythia Foundations course (link)

There's also been many interesting presentations as part of the Pangeo Showcase series (also available on YouTube). These provide insights into how people are using these tools in interesting ways for science, show the behind the scenes engineering done to build these packages, and discuss high level concepts.

If you're interested in our forum, our community calendar, want to attend our weekly community meetings, or want to present in a Pangeo Showcase - please visit our website at https://pangeo.io .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment