Skip to content

Instantly share code, notes, and snippets.

@brews
brews / intake-catalog.yaml
Last active May 8, 2021 06:36
demo workflow catalog
# Intake Catalog for the workflow in-progress
plugins:
source: [ module: intake_xarray ]
sources:
cmip6_raw:
description: Catalog of raw CMIP6 GCM data to be cleaned and prepared
metadata:
title: Raw CMIP6 GCM data
parameters:
source_id:
@brews
brews / dask-test-workflow.yaml
Last active October 31, 2023 16:22
Argo Workflow demo to launch kubernetes dask distributed job
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: dask-test-
spec:
entrypoint: dask
activeDeadlineSeconds: 1800 # Safety first, kids!
templates:
- name: dask
script:
@brews
brews / main.py
Created May 19, 2020 20:25
Example Cloud Pub/Sub cloud function to fill in missing dir on new files in gcsfuse-mounted GCS buckets
"""Fill out missing directories in GCS bucket, for gcsfuse.
Looks at blob names in GCS bucket to pull out all implied directories.
Then add a blob directory placeholder - an empty blob for each
directory that doesn't already have a blob. All directories should then
appear when mounted by gcsfuse.
"""
import logging
@brews
brews / rough_netcdf_sniffer.py
Created March 17, 2020 15:52
Sniff out odd and corrupt netCDF files throughout a directory tree.
# Parse ROOT_PATH subdirectories for netCDF files matching TARGET_GLOB,
# tries to read the file's variable, TARGET_VAR, logs any exception by printing.
import traceback
from pathlib import Path
from netCDF4 import Dataset
ROOT_PATH = "."
@brews
brews / add_encoding.py
Created February 25, 2020 02:17
Add `_Encoding` attr to netCDF4 files in subdirectories.
# Parse ROOT_PATH subdirectories for netCDF files matching TARGET_GLOB,
# if file has variable, TARGET_VAR, add "_Encoding" attribute to variable
# with VAR_ENCODING.
from pathlib import Path
from netCDF4 import Dataset
ROOT_PATH = "."
TARGET_VAR = "hierid"
@brews
brews / fillout_gcsfuse_bucket.py
Last active May 8, 2020 18:28
Python script to fill out missing directories in GCS bucket so the directory tree can be read by gcsfuse.
#! /usr/bin/env python
# 2020-01-16
# Brewster Malevich <[email protected]>
"""Fill out missing directories in GCS bucket (TARGET_BUCKET), for gcsfuse.
Looks at blob names in Google Cloud Storage bucket to pull out all implied
directories. Then add a blob directory placeholder - an empty blob for each
directory that doesn't already have a blob. All directories should then
appear when mounted by gcsfuse.
@brews
brews / lorenz96.py
Created January 16, 2020 05:17
Lorenz96 example in Python.
from scipy.integrate import odeint
import matplotlib.pyplot as plt
import numpy as np
# these are our constants
N = 40 # number of variables
F = 8 # forcing
def Lorenz96(x, t):
@brews
brews / lorenz63.py
Created January 16, 2020 05:16
Lorenz63 example in Python
import numpy as np
from scipy.integrate import odeint, solve_ivp
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
# Lorenz paramters and initial conditions
sigma, beta, rho = 10, 2.667, 28
u0, v0, w0 = 0, 1, 1.05
# Maximum time point and total number of time points
@brews
brews / backup_disks.sh
Last active December 2, 2020 21:22
Example bash script to create a basic Google Compute Engine (GCE) disk backup policy (schedule snapshot) and apply the policy to GCE disks in a zone.
#!/usr/bin/env bash
# 2020-01-02
# Create a basic GCE disk backup policy (schedule snapshot) and
# apply the policy to GCE disks in a zone.
set -e
TARGET_POLICY="backup-schedule"
ZONE="us-west1-a"
REGION="us-west1"
@brews
brews / demography_colorplot_example.R
Created September 18, 2019 00:04
Manually assign colors to series in a demography plot in burnr
# How to manually assign colors to series in a demography plot in burnr
# 2019-09-17
library(burnr)
library(ggplot2)
data(lgr2)
data(lgr2_meta)