John Nunemaker jnunemaker

Process Partitioning

The Problem

When working with large, high volume, low latency systems, it is often the case that processing data sequentially becomes detrimental to the system's health. If we only allow 1 process to work on our data we run into several challenges:

Our process may fall behind resulting in a situation which it is impossible for our process to catch up.
Our singleton process could crash and leave our system in a degraded state.
The average latency of data processing could be dramatically affected by outlying cases.

The Worker Pattern

2011-05-22

Introduction

This article is the accumulation of a tutorial that was given as a training session at Red Dirt Ruby Conf (2011) and a formal talk given at Rails Conf (2011). To understand the theory behind The Worker Pattern, read over the slides and the slide notes. From there, you can follow the instructions in the Tutorial

Get Mountain Lion and Homebrew to Be Happy

1) Install XCode 4.4 into /Applications

Get it from the App Store.

2) Install Command Line Tools

In XCode's Preferences > Downloads you can install command line tools.

ruby-1.9.3-p327 cumulative performance patch for rbenv

This installs a patched ruby 1.9.3-p327 with various performance improvements and a backported COW-friendly GC, all courtesy of funny-falcon.

Requirements

You will also need a C Compiler. If you're on Linux, you probably already have one or know how to install one. On OS X, you should install XCode, and brew install autoconf using homebrew.

	def all_replicas_in_sync?(verbose=false)
	in_sync = true
	# connect to the primary
	mongo = Mongo::Connection.new(*MongoMapper.connection.primary)
	db = mongo.db('local')
	last_op=db.collection('oplog.rs').find.sort([['$natural',-1]]).limit(1).to_a[0]
	db.collection('slaves').find().to_a.each do \|slave\|
	opdiff = last_op['ts'].increment - slave['syncedTo'].increment
	diff = (last_op['ts'].seconds - slave['syncedTo'].seconds)/1000.0
	in_sync = false if diff > 60

	#!/usr/bin/env ruby

	#
	# Annotate your "bundle outdated" with GitHub compare URLs
	#

	require 'open-uri'
	require 'json'

	def rubygems_gem_info(gem_name)

	#
	# Created by Eric Lindvall <[email protected]>
	#
	class KestrelMonitor < Scout::Plugin
	OPTIONS=<<-EOS
	host:
	label: Host
	notes: Kestrel host
	default: localhost
	port:

	require 'fileutils'

	class Reader
	def self.read(filename, processing_filename)
	return [] unless File.exists?(filename)
	FileUtils.move(filename, processing_filename)
	lines = []
	File.open(processing_filename, 'r') do \|f\|
	f.flock(File::LOCK_EX)
	while !f.eof?

	require 'rbczmq'
	require 'fileutils'
	require 'celluloid'

	class Heartbeat
	include Celluloid

	def self.start(id = nil, options = nil)
	new(ZMQ::Context.new, id, options)
	end