Matt Gillard mattgillard

A curated list of AWS resources to prepare for the AWS Certifications

A curated list of awesome AWS resources you need to prepare for the all 5 AWS Certifications. This gist will include: open source repos, blogs & blogposts, ebooks, PDF, whitepapers, video courses, free lecture, slides, sample test and many other resources.

Spark Tips & Tricks

Misc. Tips & Tricks

If values are integers in [0, 255], Parquet will automatically compress to use 1 byte unsigned integers, thus decreasing the size of saved DataFrame by a factor of 8.
Partition DataFrames to have evenly-distributed, ~128MB partition sizes (empirical finding). Always err on the higher side w.r.t. number of partitions.
Pay particular attention to the number of partitions when using flatMap, especially if the following operation will result in high memory usage. The flatMap op usually results in a DataFrame with a [much] larger number of rows, yet the number of partitions will remain the same. Thus, if a subsequent op causes a large expansion of memory usage (i.e. converting a DataFrame of indices to a DataFrame of large Vectors), the memory usage per partition may become too high. In this case, it is beneficial to repartition the output of flatMap to a number of partitions that will safely allow for appropriate partition memory sizes, based upon the

Hyperlinks (a.k.a. HTML-like anchors) in terminal emulators

[ Update 2025-03-24: Commenting is disabled permanently. Previous comments are archived at web.archive.org. ]

Most of the terminal emulators auto-detect when a URL appears onscreen and allow to conveniently open them (e.g. via Ctrl+click or Cmd+click, or the right click menu).

It was, however, not possible until now for arbitrary text to point to URLs, just as on webpages.

AWS CLI JMESPath cheatsheet

Random query recipes of JMESPath for the AWS CLI tools that I might have written or stumbled upon.

Examples

	#!/bin/bash
	# Fetch 24-hour AWS STS session token and set appropriate environment variables.
	# See http://docs.aws.amazon.com/cli/latest/reference/sts/get-session-token.html .
	# You must have jq installed and in your PATH https://stedolan.github.io/jq/ .
	# Add this function to your .bashrc or save it to a file and source that file from .bashrc .
	# https://gist.github.com/ddgenome/f13f15dd01fb88538dd6fac8c7e73f8c
	#
	# usage: aws-creds MFA_TOKEN [OTHER_AWS_STS_GET-SESSION-TOKEN_OPTIONS...]
	function aws-creds () {
	local pkg=aws-creds

	#!/bin/bash
	#
	# /etc/kernel/postinst.d script to sign akmods kmods after kernel upgrade
	#
	# Author: Michael Goodwin Date: 2016-09-21

	# 1. Copy this script to /etc/kernel/postinst.d/ and `chmod +x` it
	#
	# 2. Create signing keys (store these somewhere useful and safe):
	# $ mkdir -p /etc/pki/tls/private/mok

	# A simple cheat sheet of Spark Dataframe syntax
	# Current for Spark 1.6.1

	# import statements
	from pyspark.sql import SQLContext
	from pyspark.sql.types import *
	from pyspark.sql.functions import *

	#creating dataframes
	df = sqlContext.createDataFrame([(1, 4), (2, 5), (3, 6)], ["A", "B"]) # from manual data

	# Count total EBS based storage in AWS
	aws ec2 describe-volumes \| jq "[.Volumes[].Size] \| add"
	# Count total EBS storage with a tag filter
	aws ec2 describe-volumes --filters "Name=tag:Name,Values=CloudEndure Volume qjenc" \| jq "[.Volumes[].Size] \| add"
	# Describe instances concisely
	aws ec2 describe-instances \| jq '[.Reservations \| .[] \| .Instances \| .[] \| {InstanceId: .InstanceId, State: .State, SubnetId: .SubnetId, VpcId: .VpcId, Name: (.Tags[]\|select(.Key=="Name")\|.Value)}]'
	# Wait until $instance_id is running and then immediately stop it again
	aws ec2 wait instance-running --instance-id $instance_id && aws ec2 stop-instances --instance-id $instance_id

	# Get 10th instance in the account

	// Set up the ADAL instance, we will use the throughout the app
	export var adalInstance = new AuthenticationContext({
	instance: 'https://login.microsoftonline.com/',

	// The client ID of the app from the Azure Portal
	clientId: 'aabbccee-aabb-1122-3344-556677889900',

	// Where do we want to go after logging out
	postLogoutRedirectUri: window.location.origin,