Skip to content

Instantly share code, notes, and snippets.

View rwegener2's full-sized avatar

Rachel Wegener rwegener2

View GitHub Profile
{
"Occultation_info": {
"Occ_Date": "20200202",
"Occ_UTC": "000034Z",
"LEO_sat": {
"Receiver": "FM104",
"Source": "SPIRE"
},
"Occ_ID": "G15_FM104_20200202T000034Z", /* What goes here? */
"startTime": 126636854.0159123,
@rwegener2
rwegener2 / xesmf_debugging.ipynb
Created July 18, 2022 16:20
Example code demonstrating xesmf struggles
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rwegener2
rwegener2 / h5_cloudaccess.ipynb
Last active August 15, 2023 16:05
Accessing hdf5 data
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

Comparing v006 and v007 ATL03 Reads

Overview

When using equivalent methods the read times of v007 and v006 data are similar, with v007 being slightly faster. For both v006 and v007 data using the blocksize fsspec parameter results is a substantial speedup.

Methods

Ten v007 ATL03 files of varying file size were compared for read speeds on Cryocloud (AWS us-west-2 based Jupyter Hub) using one group of data (gt1l) using the h5py Python library. The h_ph variable as well as three relevant coordinate variables (lat_ph, lon_ph, and delta_time) were read. Times shown are the mean time of 6 independent reads. Error bars show the standard deviation of those times. All files were read 7 times. The first read was not included in the averages because it often was much slower than the subsequent reads (often 2x slower), likely due to an optimization technique from s3 that speeds up sequential reads. Additionally, each file was read 6 times consecutively prior to beginning timed reads, to decrease the effect of a