Skip to content

Instantly share code, notes, and snippets.

View alanwill's full-sized avatar
🤔
Thinking

Alan Williams alanwill

🤔
Thinking
View GitHub Profile
@alanwill
alanwill / glue-json2parquet.py
Last active July 17, 2022 07:04
AWS Glue JSON to Parquet transformation script
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
bucketpathparam = getResolvedOptions(sys.argv, ['s3_path'])
@alanwill
alanwill / transform.py
Created May 7, 2018 22:57
Copies files to a date centric directory structure based on value of a json field within the file
import os, json, shutil
from datetime import datetime
rootdir ='./input'
for subdir, dirs, files in os.walk(rootdir):
#print(subdir, dirs, files)
for file in files:
print(file)
f=open(subdir + '/'+ file,'r')
@alanwill
alanwill / verify
Last active January 4, 2017 07:23
onename verify
Verifying that "alanwill.id" is my Blockstack ID. https://onename.com/alanwill
@alanwill
alanwill / invoke_apigateway.js
Created November 10, 2016 21:12
Swagger Controller to invoke API Gateway
'use strict';
/*
'use strict' is not required but helpful for turning syntactical errors into true errors in the program flow
http://www.w3schools.com/js/js_strict.asp
*/
/*
Modules make it possible to import JavaScript files into your application. Modules are imported
using 'require' statements that give you a reference to the module.
It is a good idea to list the modules that your application depends on in the package.json in the project root
@alanwill
alanwill / delete-multiple-rows.py
Created August 21, 2016 08:59
Delete Multiple DynamoDB rows
import boto3
getCleanTable = taskStatus.query(
KeyConditionExpression=Key('requestId').eq('a3248585-8ac0-4889-b1fb-976d9113f7f2') & Key('eventTimestamp').gt('1')
)
for i in getCleanTable['Items']:
cleanupTable = boto3.client('dynamodb').batch_write_item(
RequestItems={
'talr-taskStatus': [
@alanwill
alanwill / gist:71d212d25b2061d78168ba2201187e17
Created May 11, 2016 00:08
Lambda code update with AWS CLI
aws lambda update-function-code --function my-lambda-function --zip-file fileb://lambda-package.zip
@alanwill
alanwill / api-gateway-mapping-template.json
Created April 3, 2016 06:34
AWS API Gateway Mapping Template
{
"body" : $input.json('$'),
"headers": {
#foreach($header in $input.params().header.keySet())
"$header": "$util.escapeJavaScript($input.params().header.get($header))" #if($foreach.hasNext),#end
#end
},
"method": "$context.httpMethod",
"params": {
@alanwill
alanwill / install-docker-os-engine.sh
Created February 22, 2016 19:45
Install Docker OS (Open Source) Engine on ubuntu. This assumes that `linux-image-extra-virtual` is already installed
#!/bin/bash
apt-get update -y
apt-get install apt-transport-https ca-certificates -y
apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
rm -f /etc/apt/sources.list.d/docker.list
echo "deb https://apt.dockerproject.org/repo ubuntu-trusty main" | tee /etc/apt/sources.list.d/docker.list
apt-get update -y
@alanwill
alanwill / install-docker-cs-engine1-10.sh
Last active March 8, 2017 07:03
Install latest Docker CS Engine 1.10 on ubuntu. This assumes that `linux-image-extra-virtual` is already installed.
#!/bin/bash
curl -s 'https://sks-keyservers.net/pks/lookup?op=get&search=0xee6d536cf7dc86e2d7d56f59a178ac6c6238f52e' | apt-key add --import
apt-get update -y && apt-get install apt-transport-https -y
echo "deb https://packages.docker.com/1.10/apt/repo ubuntu-trusty main" | tee /etc/apt/sources.list.d/docker.list
apt-get update -y && sudo apt-get install docker-engine -y
service docker start
usermod -a -G docker ubuntu
@alanwill
alanwill / 0_reuse_code.js
Created January 16, 2016 17:40
Here are some things you can do with Gists in GistBox.
// Use Gists to store code you would like to remember later on
console.log(window); // log the "window" object to the console