A running log of every real-world anomaly we've hit while ingesting the
s3://noaa-goes16/ABI-L2-MCMIPF/ archive into Icechunk via VirtualiZarr.
Each entry: what the file/day looks like, how it surfaced, how we handle it, and (when known) the most likely upstream cause.
When using equivalent methods the read times of v007 and v006 data are similar, with v007 being slightly faster. For both v006 and v007 data using the blocksize fsspec parameter results is a substantial speedup.
Ten v007 ATL03 files of varying file size were compared for read speeds on Cryocloud (AWS us-west-2 based Jupyter Hub) using one group of data (gt1l) using the h5py Python library. The h_ph variable as well as three relevant coordinate variables (lat_ph, lon_ph, and delta_time) were read. Times shown are the mean time of 6 independent reads. Error bars show the standard deviation of those times. All files were read 7 times. The first read was not included in the averages because it often was much slower than the subsequent reads (often 2x slower), likely due to an optimization technique from s3 that speeds up sequential reads. Additionally, each file was read 6 times consecutively prior to beginning timed reads, to decrease the effect of a
| import xarray as xr | |
| import rasterio | |
| import rioxarray | |
| import gc | |
| import earthaccess | |
| assert(earthaccess.login(strategy="netrc").authenticated) | |
| # Find a granule, load it into memory, take mean, call gc | |
| gdal_config = { |
| #!/usr/bin/env bash | |
| function p() { | |
| jq -n \ | |
| --arg content "$*" \ | |
| '{ | |
| "model": "pplx-7b-online", | |
| "messages": [ | |
| { | |
| "role": "system", | |
| "content": "Be precise and concise." |
| """ | |
| Compute and report contiguous- and chunked-dataset storage statistics for an | |
| HDF5 file that ``h5stat`` (the stock utility shipped with the HDF5 library) does | |
| not provide. | |
| Overview | |
| -------- | |
| ``h5stat`` gives a good summary of metadata, free space, and raw storage, but it | |
| stops short of the dataset-level layout details that if importance for files | |
| that use the ``PAGE`` file space strategy and are read over the network through |
| import pandas as pd | |
| import hvplot.pandas | |
| import panel as pn | |
| pn.extension(sizing_mode="stretch_width") | |
| PALETTE = [ | |
| "#ff6f69", | |
| "#ffcc5c", | |
| "#88d8b0", |
This guide is only for original Ubuntu out-of-the-box packages. If you have added a custom PPA like
pipewire-debian, you might get into conflicts.
Ubuntu 22.04 has PipeWire partially installed and enabled as it's used by browsers (WebRTC) for recoding the screeen under Wayland. We can enable remaining parts and use PipeWire for audio and Bluetooth instead of PulseAudio.
Starting from WirePlumber version 0.4.8 automatic Bluetooth profile switching (e.g. switching from A2DP to HSP/HFP when an application needs microphone access) is supported. Jammy (22.04) repos provide exactly version 0.4.8. So, we're good.
Based on Debian Wiki, but simplified for Ubuntu 22.04.
| name: CI | |
| on: | |
| push: | |
| branches: | |
| - main | |
| pull_request: | |
| branches: | |
| - main |
