@martindurant, although i've been using dask daily for a while this was my first time digging deep on a large cluster and pushing to s3. There's a lot to learn and while I don't want to hold the tools responsible for my ignorance any time tools can smooth my path i'm enormously grateful. I experienced a large stumbling block first writing with parquet. As I was very familiar with parquet locally this was somewhat easier to figure out although it was complicated by other memory issues (dask/dask#4410). Time passed and I switched to xarray and zarr. A whole new stack to get my head around. The particular error message lead me to believe I was configuring zarr incorrectly although i now understand why it's just a permissions error. I found myself switching between a few documentation pages:
Created
January 25, 2019 14:51
-
-
Save phaustin/3d322859978ae9ee6830edfc09c5ff10 to your computer and use it in GitHub Desktop.
pangeo getting started
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment