Rounak Datta rounakdatta

Context: I was asked for a list of interesting reading relating to "distributed databases, behavior under partitions and failures, failure detection." Here's what I came up with in about an hour.

For textbooks, "Introduction to Reliable and Secure Distributed Programming" is a superb introduction to distributed computing from a formal perspective; it's really not about "programming" or "engineering" but about distributed system fundamentals like consensus, distributed registers, and broadcast. Used in Berkeley's Distributed Computing course (and HT to @lalithsuresh) Book Site

Notes from courses like Lorenzo Alvisi's Distributed Computing class can be great.

There are a bunch of classics on causality, [Paxos](ht

Make it real

Ideas are cheap. Make a prototype, sketch a CLI session, draw a wireframe. Discuss around concrete examples, not hand-waving abstractions. Don't say you did something, provide a URL that proves it.

Ship it

Nothing is real until it's being used by a real user. This doesn't mean you make a prototype in the morning and blog about it in the evening. It means you find one person you believe your product will help and try to get them to use it.

Do it with style

Introduction to Kafka

Kafka acts as a kind of write-ahead log (WAL) that records messages to a persistent store (disk) and allows subscribers to read and apply these changes to their own stores in a system appropriate time-frame.

Terminology:

Producers send messages to brokers
Consumers read messages from brokers
Messages are sent to a topic

Gin, Television, and Social Surplus, or, “Looking for the Mouse”

Clay Shirky / April 26, 2008

transcription of a speech [Clay Shirky] gave at the Web 2.0 in 2008, emphasis by @jm3

I was recently reminded of some reading I did in college, way back in the last century, by a British historian arguing that the critical technology, for the early phase of the industrial revolution, was gin.

The transformation from rural to urban life was so sudden, and so wrenching, that the only thing society could do to manage was to drink itself into a stupor for a generation. The stories from that era are amazing-- there were gin pushcarts working their way through the streets of London.

And it wasn't until society woke up from that collective bender that we actually started to get the institutional structures that we associate with the industrial revolution today. Things like public libraries and museums, increasingly broad education for children, elected leaders--a lot of th

Akka Cluster Implementation Notes

Slightly disorganized but reasonably complete notes on the algorithms, strategies and optimizations of the Akka Cluster implementation. Could use a lot more links and context etc., but was just written for my own understanding. Might be expanded later.

Links to papers and talks that have inspired the implementation can be found on the 10 last pages of this presentation.

Akka Gossip

Gossip state

This is the Gossip state representation:

Jay Kreps of LinkedIn recently made an [excellent detailed 360' overview][log] post explaining why log is an underlying data structure of pretty much everything in distributed systems. Well, I feel tempted to add that same holds true for collaborative editing (like in Google Docs and others). Also, I'd like to point out the relation between partially ordered logs, the offline-first approach, AP systems and the so-called Web 3.0 (like in Meteor, pouchdb and others).

[log]: http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying The Log: What every software engineer should know about real-time data's unifying abstraction

Any real-time approach to collaborative editing decomposes a document into a sequence of atomic operations. The most known one is OT, also there is the WOOT/CRDT based family, including CT. GDocs is an OT system. I did a CT system currently in beta at http://letters.yandex.ru. Note that letters is a tiny pilot project

	#!/usr/bin/env python
	# -- coding: utf-8 --
	#
	# fetch_and_combine.py
	#

	"""
	Scans CloudFront logs in an S3 bucket for any that are new. Combines log files
	into a single local file per hour. If logs for multiple CloudFront
	distributions are present, combines them all.

	# undo: reverts the last command, as long as there is a registered handler
	# for it.
	#
	# Usage:
	#
	# $ undo
	#
	# There are no options.
	#
	# There are by default registered handlers for the following commands:

	import com.amazonaws.auth.BasicAWSCredentials
	import com.amazonaws.services.s3.AmazonS3Client
	import com.amazonaws.AmazonClientException
	import com.amazonaws.AmazonServiceException
	import java.io.File
	import java.io.BufferedReader
	import java.io.InputStreamReader
	import java.io.FileOutputStream
	import org.apache.commons.io.IOUtils

	#!flask/bin/python
	from flask import Flask, jsonify, abort, request, make_response, url_for
	from flask_httpauth import HTTPBasicAuth

	app = Flask(__name__, static_url_path = "")
	auth = HTTPBasicAuth()

	@auth.get_password
	def get_password(username):
	if username == 'miguel':