Skip to content

Instantly share code, notes, and snippets.

View TomNicholas's full-sized avatar

Tom Nicholas TomNicholas

View GitHub Profile
@TomNicholas
TomNicholas / ConcatenatableArray.ipynb
Created February 4, 2024 21:14
Try to make a minimal ConcatenatableArray class work with xarray
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@TomNicholas
TomNicholas / dim_coord_test.ipynb
Last active March 7, 2024 17:53
Kerchunk non-deterministic bugs
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@TomNicholas
TomNicholas / test_icechunk_refs_at_scale.ipynb
Created November 20, 2024 19:59
Testing writing huge amounts of virtual references into icechunk
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@TomNicholas
TomNicholas / oae_refs_at_scale.ipynb
Created January 30, 2025 18:52
VirtualiZarr'ing CWorthy OAE dataset
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@TomNicholas
TomNicholas / benchmark_hdf.py
Last active April 9, 2025 19:30
Benchmark reading netCDF4 file via HTTP GET requests from S3
# /// script
# requires-python = ">=3.13"
# dependencies = [
# "requests",
# "h5py",
# "xarray",
# "h5netcdf",
# ]
# ///
@TomNicholas
TomNicholas / benchmark.py
Created August 8, 2025 07:43
xarray .to_zarr scaling benchmarks
# /// script
# requires-python = ">=3.13"
# dependencies = [
# "xarray",
# "zarr",
# "matplotlib",
# ]
# ///
import asyncio

GOES-16 ABI-L2-MCMIPF data inconsistencies

A running log of every real-world anomaly we've hit while ingesting the s3://noaa-goes16/ABI-L2-MCMIPF/ archive into Icechunk via VirtualiZarr.

Each entry: what the file/day looks like, how it surfaced, how we handle it, and (when known) the most likely upstream cause.