Skip to content

Instantly share code, notes, and snippets.

View douglascodes's full-sized avatar

Douglas King douglascodes

View GitHub Profile
@douglascodes
douglascodes / clean_iostat.awk
Created June 29, 2018 15:47
Awk script for taking grouped drives and outputting single line with current time. Pipe to text log
#!/usr/bin/env awk
# iostat -t -d -H -g hdds sda sdb
{OFS="\t"}
NR==3 { runtime=$1;}
NR==5 { print runtime, $1, $2, $3, $4, $5, $6}
@douglascodes
douglascodes / gather_hdd_stats.sh
Created June 29, 2018 15:49
Creates a log file for iostat data
#! /usr/bin/env bash
# gather_hdd_stats.sh $(date +%F).log
# Every 5 seconds on the mod 5 pipe formatted iostat data to <file>
if [ $# -lt 1 ] ; then
echo "usage: $0 <file>"
exit -1
fi
outputfile="$1"
@douglascodes
douglascodes / recamán.py
Created July 2, 2018 04:43
Recamán sequence simple.
import sys
from collections import Counter
def main():
next_location = 0
history = Counter()
for i in range(1, 100):
print(next_location)
next_location = racaman(next_location, i, history)
@douglascodes
douglascodes / pandas_apply_two_column_result.py
Created July 31, 2019 17:50
Example for assign Tuple result to two columns after .apply() in Pandas
import pandas as pd
import random
def avg(a, b):
return (a+b)/2
def diff(a, b):
return abs(a-b)
def get_stats(a, b):
@douglascodes
douglascodes / glue_script.py
Last active January 6, 2021 18:56
AN AWS Glue script for remote debugging example
# For https://support.wharton.upenn.edu/help/glue-debugging
import uuid
from pyspark.sql.functions import *
from pyspark.sql.types import StringType, IntegerType, StructField, StructType
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions
import sys
from pyspark.context import SparkContext
import datetime
@douglascodes
douglascodes / aws_var_encryption.py
Last active May 4, 2020 20:31
Encrypts all environment variables using the KMS key specified in env['aws_key_arn']. Much simpler to use as a utility in something like PyCharm with run configurations.
#! /usr/bin/python
"""
Script to encrypt environment variables to their encrypted version.
aws_key_arn - Resource ID # for aws encryption key.
format: 'arn:aws:kms:us-east-1:123456789:key/711d0e6d-620c-47da-a2f6-7141eb8cbde4'
"""
import os
import boto3
from base64 import b64encode