A note on log messages

scratch buffer

abacus.go

	Log Messages

		All of our production systems in The Vault have their
		STDOUT connected to Heroku's logplex. We also connect
		syslog draings to Logplex that outlet all messages to
		PaperTrailapp. Each system outputs two classes of
		messages. The first class of message is the system
		class. Messages of the system class are originated by
		libraries and low-level components. Compenets like
		queueing libraries, application frameworks, http servers,
		etc.... The second class of message originates from our
		application's code base. These messages are crafted by the
		engineer of the application. Application messages
		generally describe how our application is behaving in
		realtime. Think of these messages as tracer bullets,
		realtime comments that descibe how our system is
		processing data. For example:

			Oct 15 11:15:17 vault app/web.1:  account=42641 #begin_authorization method=bt_authorize
			Oct 15 11:15:17 vault app/web.1:  account=42641 method=build_result verify=true
			Oct 15 11:15:17 vault app/web.1:  account=42641 method=create_card verify_card=true
			Oct 15 11:15:17 vault app/web.1:  [Braintree] [15/Oct/2011 18:15:17 UTC] POST /customers 422
			Oct 15 11:15:17 vault app/web.1:  account=42641 #failed_authorization  errors=Credit card number is required.

		You will notice that the all but one message listed
		above deals with application specfic content.

		Since all of our messages will eventually end up in
		PaperTrailapp, we can now consider building  auxiliary
		systems that will give us unbelievable visibility into
		our production systems. PaperTrailapp gives us a query
		interface so that we can do quick searches and debug
		runtime scenarios. Consider searching for
		"account=42641."  Such a query would produce results
		similar to those listed above resutling in trivial
		debugging when something has gone wrong with account
		42641. But there are certain types of messages that we
		want to be notified about. Instead of querying for
		data after the fact, we want to setup some sort of
		system that reacts to the message in real-time.

		The previous paragraphs lead us up to a module of
		code. The module will act as the outlet to
		PaperTrailapp. Our module will react
		to messages that are matched by some query in
		PaperTrailapp and subsequently forwared.


	Subject Counter
		
		There will be infinitely many function we wish to
		apply to our stream of data that comes from
		PaperTrailapp. One such function is counting the
		number of unique message subjects. A message subject
		is a special sub-string in our message payload. You
		will notice in figure 1.1 that several sub-strings
		start with a '#' character. You can think of these
		sub-strings as Twitter hash-tags for logs. For example, 
		I may wish to know how many times an account
		begins authoriztaion and how many times an account
		fails_authorization. To acquire this knowledge, we
		will count the number of times we see the respective
		subject in our stream of data.

		I have found that it can be useful to treat
		log data as ephemeral. This implies that if we want to
		leverage a subset of our log data over time, we should come up
		with some sort of durability story for that
		subset. For our subject counters, we will want to
		persist our coutners to some sort of durable
		system. PostgreSQL is one such system.