Skip to content

Instantly share code, notes, and snippets.

View benfasoli's full-sized avatar

Ben Fasoli benfasoli

View GitHub Profile
@benfasoli
benfasoli / limit_asyncio_concurrency_example.py
Created January 23, 2022 17:18
Limit concurrency with Python asyncio
import asyncio
from typing import Coroutine, List, Sequence
def _limit_concurrency(
coroutines: Sequence[Coroutine], concurrency: int
) -> List[Coroutine]:
"""Decorate coroutines to limit concurrency.
Enforces a limit on the number of coroutines that can run concurrently in higher
@benfasoli
benfasoli / map_spatial_average.r
Created March 26, 2021 15:34
Make interactive map by spatially averaging high resolution observations
#!/usr/bin/env Rscript
options(dplyr.summarise.inform = F)
library(data.table)
library(htmlwidgets)
library(leaflet)
library(tidyverse)
library(ggplot2)
library(metR)
time <- Sys.time()
n <- 50
data <- data.frame(
time = seq(time - 3600, time, length.out = n),
direction = seq(0, 359, length.out = n),
magnitude = 5 * sin(seq(0, 2 * pi, length.out = n))
)
@benfasoli
benfasoli / pandas_read_csv_from_string.py
Last active October 28, 2020 23:42
Example of `io.StringIO` to pass text to `pd.read_csv(text)`
#!/usr/bin/env python3
import io
import pandas as pd
api_response_csv = """
x, y, z
1, 2, 3
@benfasoli
benfasoli / scluster
Last active October 20, 2020 19:58
CHPC SLURM convenience utilities
#!/bin/bash
set -e
if [[ "$HOSTNAME" == "notchpeak"* ]]; then
CLUSTER="np"
elif [[ "$HOSTNAME" == "kingspeak"* ]]; then
CLUSTER="kp"
else
echo "Unable to determine cluster from hostname: $HOSTNAME" > /dev/stderr
exit 1
@benfasoli
benfasoli / run_test.r
Created October 9, 2020 19:43
Test exit handling of forked processes managed by R's parallel library
#!/usr/bin/env Rscript
# Ben Fasoli | [email protected]
# Return number of R processes currently running for active user
get_process_count <- function(x = NA) {
as.integer(system('pgrep -U $USER R | wc -l', intern = T))
}
N_FORKS <- 2
@benfasoli
benfasoli / read_mesowest.py
Created June 24, 2020 20:16
Bulk download from MesoWest API and use BigQuery streaming API to load into table
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from datetime import datetime, timedelta
from multiprocessing import cpu_count, Pool
import os
from typing import List
from google.cloud import bigquery
import pandas as pd
@benfasoli
benfasoli / read_traj.py
Last active June 12, 2020 21:50
Read STILT particle trajectories from python
#!/usr/bin/env python3
import io
import os
import subprocess
import pandas as pd
class ReadTrajException(Exception):
#!/usr/bin/env Rscript
library(raster)
library(tidyverse)
# Time integrate footprints to a single layer
r0 <- sum(brick('footprint0.nc'))
r1 <- sum(brick('footprint1.nc'))
control <- values(r0)
#!/bin/bash
EXE=./hycs_std
OUTPUT=PARTICLE.DAT
COUNT=0
while ! grep -q NaN $OUTPUT; do
$EXE
let COUNT=COUNT+1
done