Rounak Datta rounakdatta

Akka Cluster Implementation Notes

Slightly disorganized but reasonably complete notes on the algorithms, strategies and optimizations of the Akka Cluster implementation. Could use a lot more links and context etc., but was just written for my own understanding. Might be expanded later.

Links to papers and talks that have inspired the implementation can be found on the 10 last pages of this presentation.

Akka Gossip

Gossip state

This is the Gossip state representation:

Gin, Television, and Social Surplus, or, “Looking for the Mouse”

Clay Shirky / April 26, 2008

transcription of a speech [Clay Shirky] gave at the Web 2.0 in 2008, emphasis by @jm3

I was recently reminded of some reading I did in college, way back in the last century, by a British historian arguing that the critical technology, for the early phase of the industrial revolution, was gin.

The transformation from rural to urban life was so sudden, and so wrenching, that the only thing society could do to manage was to drink itself into a stupor for a generation. The stories from that era are amazing-- there were gin pushcarts working their way through the streets of London.

And it wasn't until society woke up from that collective bender that we actually started to get the institutional structures that we associate with the industrial revolution today. Things like public libraries and museums, increasingly broad education for children, elected leaders--a lot of th

Introduction to Kafka

Kafka acts as a kind of write-ahead log (WAL) that records messages to a persistent store (disk) and allows subscribers to read and apply these changes to their own stores in a system appropriate time-frame.

Terminology:

Producers send messages to brokers
Consumers read messages from brokers
Messages are sent to a topic

Make it real

Ideas are cheap. Make a prototype, sketch a CLI session, draw a wireframe. Discuss around concrete examples, not hand-waving abstractions. Don't say you did something, provide a URL that proves it.

Ship it

Nothing is real until it's being used by a real user. This doesn't mean you make a prototype in the morning and blog about it in the evening. It means you find one person you believe your product will help and try to get them to use it.

Do it with style

Context: I was asked for a list of interesting reading relating to "distributed databases, behavior under partitions and failures, failure detection." Here's what I came up with in about an hour.

For textbooks, "Introduction to Reliable and Secure Distributed Programming" is a superb introduction to distributed computing from a formal perspective; it's really not about "programming" or "engineering" but about distributed system fundamentals like consensus, distributed registers, and broadcast. Used in Berkeley's Distributed Computing course (and HT to @lalithsuresh) Book Site

Notes from courses like Lorenzo Alvisi's Distributed Computing class can be great.

There are a bunch of classics on causality, [Paxos](ht

In August 2007 a hacker found a way to expose the PHP source code on facebook.com. He retrieved two files and then emailed them to me, and I wrote about the issue:

http://techcrunch.com/2007/08/11/facebook-source-code-leaked/

It became a big deal:

http://www.techmeme.com/070812/p1#a070812p1

The two files are index.php (the homepage) and search.php (the search page)

	#!flask/bin/python
	from flask import Flask, jsonify, abort, request, make_response, url_for
	from flask_httpauth import HTTPBasicAuth

	app = Flask(__name__, static_url_path = "")
	auth = HTTPBasicAuth()

	@auth.get_password
	def get_password(username):
	if username == 'miguel':

	import com.amazonaws.auth.BasicAWSCredentials
	import com.amazonaws.services.s3.AmazonS3Client
	import com.amazonaws.AmazonClientException
	import com.amazonaws.AmazonServiceException
	import java.io.File
	import java.io.BufferedReader
	import java.io.InputStreamReader
	import java.io.FileOutputStream
	import org.apache.commons.io.IOUtils

	# undo: reverts the last command, as long as there is a registered handler
	# for it.
	#
	# Usage:
	#
	# $ undo
	#
	# There are no options.
	#
	# There are by default registered handlers for the following commands:

	#!/usr/bin/env python
	# -- coding: utf-8 --
	#
	# fetch_and_combine.py
	#

	"""
	Scans CloudFront logs in an S3 bucket for any that are new. Combines log files
	into a single local file per hour. If logs for multiple CloudFront
	distributions are present, combines them all.