douglascodes’s gists

douglascodes / aws_var_encryption.py

Last active May 4, 2020 20:31

Encrypts all environment variables using the KMS key specified in env['aws_key_arn']. Much simpler to use as a utility in something like PyCharm with run configurations.

	#! /usr/bin/python
	"""
	Script to encrypt environment variables to their encrypted version.
	aws_key_arn - Resource ID # for aws encryption key.
	format: 'arn:aws:kms:us-east-1:123456789:key/711d0e6d-620c-47da-a2f6-7141eb8cbde4'
	"""
	import os
	import boto3
	from base64 import b64encode

douglascodes / glue_script.py

Last active January 6, 2021 18:56

AN AWS Glue script for remote debugging example

	# For https://support.wharton.upenn.edu/help/glue-debugging
	import uuid
	from pyspark.sql.functions import *
	from pyspark.sql.types import StringType, IntegerType, StructField, StructType
	from awsglue.context import GlueContext
	from awsglue.job import Job
	from awsglue.utils import getResolvedOptions
	import sys
	from pyspark.context import SparkContext
	import datetime

douglascodes / pandas_apply_two_column_result.py

Created July 31, 2019 17:50

Example for assign Tuple result to two columns after .apply() in Pandas

	import pandas as pd
	import random

	def avg(a, b):
	return (a+b)/2

	def diff(a, b):
	return abs(a-b)

	def get_stats(a, b):

douglascodes / recamán.py

Created July 2, 2018 04:43

Recamán sequence simple.

	import sys
	from collections import Counter


	def main():
	next_location = 0
	history = Counter()
	for i in range(1, 100):
	print(next_location)
	next_location = racaman(next_location, i, history)

douglascodes / gather_hdd_stats.sh

Created June 29, 2018 15:49

Creates a log file for iostat data

	#! /usr/bin/env bash
	# gather_hdd_stats.sh $(date +%F).log
	# Every 5 seconds on the mod 5 pipe formatted iostat data to <file>
	if [ $# -lt 1 ] ; then
	echo "usage: $0 <file>"
	exit -1
	fi

	outputfile="$1"

douglascodes / clean_iostat.awk

Created June 29, 2018 15:47

Awk script for taking grouped drives and outputting single line with current time. Pipe to text log

	#!/usr/bin/env awk
	# iostat -t -d -H -g hdds sda sdb
	{OFS="\t"}
	NR==3 { runtime=$1;}
	NR==5 { print runtime, $1, $2, $3, $4, $5, $6}

douglascodes / aws_redhat_basic.yml

Last active September 17, 2017 21:46

An ansible file for setting up my default config

	---
	- hosts: redhat_aws
	vars:
	remote_user: ec2-user
	target_user: myUserName
	tasks:
	- name: 1.Check if EPEL repo is already configured.
	stat: path=/etc/yum.repos.d/epel.repo
	register: epel_repofile_result

douglascodes / inotify_sync.sh

Last active July 5, 2017 11:21

Script to update web directory on change. A -> B Watch using inotifywait.

	#!/bin/sh
	# Syncs files from directory A to B. With the 'chown' this will need to be run as root.
	# Drop that to run as unprivileged user.

	if [ $# -lt 2 ] ; then
	echo "usage: $0 <dir to watch> <sync to>"
	fi

	source="$1"
	destination="$2"

douglascodes / getDaysOldFiles.ps1

Created February 23, 2017 02:40

Gets files where -14 days old. Can be turned into a function easily.

Get-ChildItem -Recurse  | Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-14) }

douglascodes / removeOldBackups.ps1

Last active February 23, 2017 02:36

Function for testing files in similar paths but different roots exist in both places

	# Script for testing file existence in two places
	# PS C:\> . .\removeOldBackups.ps1
	# PS C:\> existsInBoth -DestRoot 'C:\Program Files\' -SourceRoot 'C:\Program Files (x86)\' -CommonPath "Common Files"
	# True
	# PS C:\> existsInBoth -DestRoot 'C:\Program Files\' -SourceRoot 'C:\Program Files (x86)\' -CommonPath "UnmatchedFile"
	# False

	function existsInBoth{
	param ( [string] $SourceRoot,
	[string] $DestRoot,

Douglas King douglascodes