Vince Tse vincetse

Deep Dive: PostgreSQL on AWS

When loading data

disable backups (backup_retention=0)
disable multi-AZ and autovacuum
pg_dump -Fc (compressed) and pg_restore -j (parallel)
Increase maintenance_work_mem

AWS CLI JMESPath cheatsheet

Random query recipes of JMESPath for the AWS CLI tools that I might have written or stumbled upon.

Examples

Installing cloud-init on a fresh Raspbian Lite image

This is a work in Progress!

Purpose

This mainly demonstrates my goal of preparing a Raspberry Pi to be provisioned prior to its first boot. To do this I have chosen to use the same cloud-init that is the standard for provisioning servers at Amazon EC2, Microsoft Azure, OpenStack, etc.

I found this to be quite challenging because there is little information available for using cloud-init without a cloud. So, this project also servers as a demonstration for anyone on any version of Linux who may want to install from source, and/or use without a cloud. If you fall into that later group, you probably just want to read the code. It's bash so everything I do, you could also do at the command line. (Even the for loop.)

	Following the news about Facebook buying Instagram I decided to delete my Instagram account before Facebook claims ownership of my pictures.
	Since the Instagram-recommended (in their FAQ): http://instaport.me/export doesn't work for me (probably they can't cope with the high demand),
	here is a quick and dirty way to download all my Instagram pictures in their highest resolution in a few easy steps.

	You will need: Firefox, Firebug, some text editor, wget

	1. Go to http://statigr.am/yourlogin using Firefox with Firebug extension active
	2. Scroll down as many times as it is needed to have all yor pictures thumbnails displayed (I had some 3 hundred pictures so it was not that much scrolling, YMMV)
	3. In the Firebug JS console run this JS code: $(".lienPhotoGrid a img").each(function(index) { console.log($(this).attr('src')) })
	4. JS console will contain urls to all the thumbnails images, like this: http://distilleryimage1.s3.amazonaws.com/4ed46cf2801511e1b9f1123138140926_5.jpg

	import java.util.Properties
	import kafka.server.KafkaServer
	import kafka.server.KafkaConfig
	import kafka.producer.ProducerConfig
	import kafka.producer.Producer
	import kafka.message.Message
	import kafka.producer.ProducerData
	import kafka.consumer.ConsumerConfig
	import kafka.consumer.Consumer
	import kafka.utils.Utils

	#!/bin/bash
	#
	# this script will attempt to detect any ephemeral drives on an EC2 node and create a RAID-0 stripe
	# mounted at /mnt. It should be run early on the first boot of the system.
	#
	# Beware, This script is NOT fully idempotent.
	#

	METADATA_URL_BASE="http://169.254.169.254/2012-01-12"

	package main

	import (
	"fmt"
	//"bytes"
	//"strconv"
	)

	type AVLTree struct {
	Val int

	import logging
	import logging.config
	import sumologger
	from sumologger import SumoHTTPHandler

	logging.config.dictConfig(sumologger.LOGGING)
	logger = logging.getLogger("sumologger")
	logger.debug("Nifty log message")

	# -- mode: ruby --
	# vi: set ft=ruby :

	CLOUD_CONFIG_PATH = File.join(File.dirname(__FILE__), "user-data")

	# All Vagrant configuration is done below. The "2" in Vagrant.configure
	# configures the configuration version (we support older styles for
	# backwards compatibility). Please don't change it unless you know what
	# you're doing.