pbrumblay’s gists

pbrumblay / test-local-redis.mjs

Created November 1, 2021 22:56

	import Redis from 'ioredis';

	const redis = new Redis({
	keyPrefix: "hereIsAPrefixToNamespaceThings:",
	port: 6379,
	host: 'localhost',
	db: 0,
	connectTimeout: 10000,
	});

pbrumblay / google_cloud_helper.py

Last active November 10, 2020 18:50

Using Tenacity for retries

	""" gcloud storage helper functions """
	import json
	import logging
	import posixpath
	from typing import Tuple, Any, Dict, List

	from tenacity import retry, stop_after_attempt, wait_exponential
	from google.cloud import storage

	storage_client = storage.Client()

pbrumblay / AUD_CAD.csv

Created August 13, 2020 14:03

Sample data for https://gist.github.com/pbrumblay/b8fee2a425be6bfc88fc43ff15fb0199

We can make this file beautiful and searchable if this error is corrected: It looks like row 9 should actually have 14 columns, instead of 1 in line 8.

	2020-04-20 21:00:00+00:00,0.89533,0.89664,0.88963,0.89218,0.89733,0.89733,0.88993,0.89287,0.89633,0.89677,0.8898,0.89252,236284
	2020-04-21 21:00:00+00:00,0.89215,0.89983,0.89112,0.89502,0.8929,0.90124,0.89136,0.89557,0.89252,0.9005,0.89125,0.8953,177029
	2020-04-22 21:00:00+00:00,0.8946,0.90112,0.89169,0.89633,0.89604,0.90164,0.89202,0.89676,0.89532,0.90128,0.89187,0.89654,187587
	2020-04-23 21:00:00+00:00,0.89608,0.902,0.89324,0.90008,0.89722,0.9027,0.89365,0.90208,0.89665,0.90225,0.89346,0.90108,149012
	2020-04-26 21:00:00+00:00,0.899,0.90973,0.89892,0.90706,0.901,0.91003,0.9009,0.90744,0.9,0.90982,0.89992,0.90725,146455
	2020-04-27 21:00:00+00:00,0.90646,0.91077,0.90363,0.90832,0.9077,0.91095,0.90445,0.9088,0.90708,0.91086,0.90426,0.90856,153814
	2020-04-28 21:00:00+00:00,0.90823,0.91213,0.90704,0.90972,0.90915,0.91235,0.9075,0.91017,0.90869,0.91222,0.90742,0.90994,130235
	2020-04-29 21:00:00+00:00,0.90988,0.9113,0.90253,0.90792,0.91032,0.91162,0.90282,0.90843,0.9101,0.91146,0.90269,0.90818,191082
	2020-04-30 21:

pbrumblay / add_row.py

Last active September 29, 2020 04:34

Add row to pandas_ta dataframe and recompute

	import pandas as pd
	import pandas_ta as ta
	from dateutil import parser

	df = pd.read_csv('AUD_CAD.csv', sep=',', names=[
	'datetime', 'bid_open', 'bid_high', 'bid_low', 'bid_close', 'ask_open', 'ask_high', 'ask_low', 'ask_close', 'mid_open', 'mid_high', 'mid_low', 'mid_close', 'volume'], error_bad_lines=False, parse_dates=['datetime'])

	df.ta.atr(append=True, high='bid_high', low='bid_low', close='bid_close')

	print(df)

pbrumblay / terraform_plan_debug.txt

Created January 3, 2020 21:37

Upgrade settings block not working

	terraform plan
	2020/01/03 14:33:49 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
	Use TF_LOG=TRACE to see Terraform's internal logs.
	----
	2020/01/03 14:33:49 [INFO] Terraform version: 0.12.18
	2020/01/03 14:33:49 [INFO] Go runtime version: go1.12.13
	2020/01/03 14:33:49 [INFO] CLI args: []string{"/usr/local/bin/terraform", "plan"}
	2020/01/03 14:33:49 [DEBUG] Attempting to open CLI config file: /Users/peter/.terraformrc
	2020/01/03 14:33:49 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
	2020/01/03 14:33:49 [INFO] CLI command args: []string{"plan"}

pbrumblay / gcs_custom_hook.py

Created November 5, 2018 22:16

Airflow custom Google Cloud Storage Hook with resumable uploads, partial downloads, and compose (everyone else calls it "concatenating") functionality

	from google.cloud import storage
	from airflow.hooks.base_hook import BaseHook
	from airflow.utils.log.logging_mixin import LoggingMixin
	import random
	import string


	class GCSCustomHook(BaseHook, LoggingMixin):

	def __init__(self, storage_conn_id='google_cloud_storage_default'):

pbrumblay / dump_avro_schema.py

Created July 13, 2018 14:28

Python script to extract schema from avro file in google cloud storage

	from google.cloud import storage
	import sys
	from avro.datafile import DataFileReader
	from avro.io import DatumReader
	import json

	client = storage.Client()

	bucket_name = sys.argv[1]
	blob_name = sys.argv[2]

pbrumblay / GetTableShardPartition.java

Last active June 4, 2018 13:54

Table destination function which supports large numbers of partitions

	import com.google.api.services.bigquery.model.TableRow;
	import com.google.api.services.bigquery.model.TimePartitioning;
	import org.apache.beam.sdk.io.gcp.bigquery.TableDestination;
	import org.apache.beam.sdk.transforms.SerializableFunction;
	import org.apache.beam.sdk.values.ValueInSingleWindow;
	import org.slf4j.Logger;
	import org.slf4j.LoggerFactory;

	//From: https://shinesolutions.com/2017/12/05/fun-with-serializable-functions-and-dynamic-destinations-in-cloud-dataflow/
	public class GetTableShardPartition implements SerializableFunction<ValueInSingleWindow<TableRow>, TableDestination> {

pbrumblay / ReplaceBQPartition.java

Created May 25, 2018 18:01

Use BigQueryIO.writeTableRows() to replace partitions based off values in TableRow elements

	package com.fearlesstg;

	import com.google.api.services.bigquery.model.TableFieldSchema;
	import com.google.api.services.bigquery.model.TableRow;
	import com.google.api.services.bigquery.model.TableSchema;
	import com.google.api.services.bigquery.model.TimePartitioning;
	import org.apache.beam.sdk.Pipeline;
	import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO;
	import org.apache.beam.sdk.io.gcp.bigquery.TableDestination;
	import org.apache.beam.sdk.options.PipelineOptions;

pbrumblay / ManualLoad.java

Created May 25, 2018 17:14

Use low level com.google.api.services.bigquery.Bigquery client to create load job to truncate partition

	package com.fearlesstg;

	import com.google.api.client.googleapis.auth.oauth2.GoogleCredential;
	import com.google.api.client.http.HttpTransport;
	import com.google.api.client.http.javanet.NetHttpTransport;
	import com.google.api.client.json.JsonFactory;
	import com.google.api.client.json.jackson.JacksonFactory;
	import com.google.api.services.bigquery.Bigquery;
	import com.google.api.services.bigquery.BigqueryScopes;
	import com.google.api.services.bigquery.model.*;

Peter Brumblay pbrumblay