Matt Weyant mattweyant

Introduction to Kafka

Kafka acts as a kind of write-ahead log (WAL) that records messages to a persistent store (disk) and allows subscribers to read and apply these changes to their own stores in a system appropriate time-frame.

Terminology:

Producers send messages to brokers
Consumers read messages from brokers
Messages are sent to a topic

Tuning Storm+Trident

Tuning a dataflow system is easy:

The First Rule of Dataflow Tuning:
* Ensure each stage is always ready to accept records, and
* Deliver each processed record promptly to its destination

	# ElasticSearch Service

	description "ElasticSearch"

	start on (net-device-up
	and local-filesystems
	and runlevel [2345])

	stop on runlevel [016]

	####################################
	# BASIC REQUIREMENTS
	# http://graphite.wikidot.com/installation
	# http://geek.michaelgrace.org/2011/09/how-to-install-graphite-on-ubuntu/
	# Last tested & updated 9/6/2013
	####################################

	cd
	sudo apt-get update
	sudo apt-get upgrade

	# All credit: http://stackoverflow.com/questions/4006324/how-to-atomically-delete-keys-matching-a-pattern-with-redis
	redis-cli [options] KEYS "prefix:*" \| xargs redis-cli [options] DEL