Steve Coffman StevenACoffman

Links to live by

Networking

Fundamentals

TCP Summary (including three-way handshake description): https://condor.depaul.edu/jkristof/technotes/tcp.html
TCP Headers: http://www.omnisecu.com/tcpip/tcp-header.php
TCP full protocol: http://www.networksorcery.com/enp/protocol/tcp.htm

Network management on Linux

http://www.cyberciti.biz/faq/setting-up-an-network-interfaces-file/

pRPC: gRPC on Classic AppEngine today

gRPC is great, but is not available on Classic AppEngine at this time, so while working on the [nextgen CI for Chromium][luci] we wrote pRPC (provisional RPC) for Go that is compatible with gRPC server/client code, but works on HTTP 1.x and Classic AppEngine. In addition it has gRPC-compatible discovery and a CLI tool.

IDL

Experimenting with AWS Lambda for ETL

A lot of us are interested in doing more analysis with our service logs so I thought I'd share an experiment I'm doing with Sync. The main idea is to transform the raw logs into something that'll be nice to query and generate reports with in Redshift.

The Pipeline

Logs make their way into an S3 bucket (lets call it the 'raw' bucket) where we've got a lambda listening for new data. This lambda reads the raw heka protobuf gzipped data, does some transformation and writes a new file to a different S3 bucket (the 'processed' bucket) in a format that is redshift friendly (like json or csv). There's another lambda listening on the processed bucket that loads this data into Redshift.

	"""Simple demo of using Flask with aiohttp via aiohttp-wsgi's
	WSGIHandler.

	"""

	import asyncio
	from aiohttp import web
	from aiohttp_wsgi import WSGIHandler
	from flask import Flask, render_template

	#!/bin/bash

	set -e -x

	yum -y update
	yum -y upgrade
	yum -y groupinstall "Development Tools"
	yum -y install zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel git

	export HOME=/home/ec2-user

	# Copyright (c) Productize
	# This is free software: you can redistribute it and/or modify
	# it under the terms of the GNU General Public License version 3
	# as published by the Free Software Foundation.
	# You can get the license at <http://www.gnu.org/licenses/gpl.html>
	#
	# This program is distributed in the hope that it will be useful,
	# but WITHOUT ANY WARRANTY; without even the implied warranty of
	# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
	# GNU General Public License for more details.

	# This is a "Managed Script" in Jenkins
	COMMIT=`aws lambda get-alias --region $AWS_REGION --function-name $FUNCTION_NAME --name $PUBLISH_FROM_ALIAS \| grep "Description" \| cut -d'"' -f4`
	VERSION=`aws lambda publish-version --region $AWS_REGION --function-name $FUNCTION_NAME --description $COMMIT \| grep "Version" \| cut -d'"' -f4`
	aws lambda update-alias --region $AWS_REGION --function-name $FUNCTION_NAME --function-version $VERSION --name $PUBLISH_TO_ALIAS --description $COMMIT

	package main

	import (
	"github.com/apex/go-apex"
	"github.com/apex/go-apex/kinesis"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/firehose"
	)

	const mapping = (f) => (reducing) => (result, input) => reducing(result, f(input));
	const filtering = (predicate ) => (reducing) => (result, input) => predicate(input) ? reducing(result, input) : result;

	import socket, machine

	# This is the HTML, CSS and js to deliver to the client.
	# You can (should) store this in another file and read from there,
	# but I am lazy.
	# The CSS and HTML to display pretty code is from pygments.
	upy = """<!DOCTYPE html>
	<html>
	<head>
	<title>I'm a website running on a Microcontroller.</title>