weijh imweijh

Long story, short: I'm totally open to supporting more rubies if possible. Details follow.

Related issue: http://code.google.com/p/logstash/issues/detail?id=37

Summary:

core and stdlib ruby changes violently and without notice and without backwards compatibility. I want nothing of that.
need a cross-ruby date library that isn't part of stdlib (see previous point) and is also good.
need an easy way to use multiple cpus that is cross-ruby (threads are not it)

Logstash internals (Queues and Threading)

The logstash agent is 3 parts: inputs -> filters -> outputs.

Each '->' is an internal messaging system. It is implemented with a 'SizedQueue' in Ruby. SizedQueue allows a bounded maximum of items in the queue such that any writes to the queue will block if the queue is full at maximum capacity.

Logstash sets the queue size to 20. This means only 20 events can be pending into the next phase - this helps reduce any data loss and in general avoids logstash trying to act as a data storage system. These internal queues are not for storing messages long-term.

In reverse, here's what happens with a queue fills.

Latency numbers every programmer should know

L1 cache reference ......................... 0.5 ns
Branch mispredict ............................ 5 ns
L2 cache reference ........................... 7 ns
Mutex lock/unlock ........................... 25 ns
Main memory reference ...................... 100 ns             
Compress 1K bytes with Zippy ............. 3,000 ns  =   3 µs
Send 2K bytes over 1 Gbps network ....... 20,000 ns  =  20 µs
SSD random read ........................ 150,000 ns  = 150 µs

Read 1 MB sequentially from memory ..... 250,000 ns = 250 µs

tmux shortcuts & cheatsheet

start new:

tmux

start new with session name:

tmux new -s myname

logstash needs full time love.

There are roughly 70 code contributors to the logstash project as of today. Many more exist as helpful folks on IRC and the mailling list. Others post awesome bug reports and feature requests. The community is simply awesome.

I've been working on logstash for about 2 years. All as a hobby - all part-time. About 70% of the current content (commits, 'git blame' lines, etc) are by me.

Unfortunately, I've never used logstash in production. (Embarrassing, I know!)

I'm embarrassed to answer user questions like "How do I optimize elasticsearch's storage?" with "Sorry, I've never used elasticsearch outside of my laptop."

	#rsyslog v3 config file

	# if you experience problems, check
	# http://www.rsyslog.com/troubleshoot for assistance

	#### MODULES ####

	# Provides UDP syslog reception
	$ModLoad imudp.so
	$UDPServerRun 514

	# the following two lines give a two-line status, with the current window highlighted
	hardstatus alwayslastline
	hardstatus string '%{= kG}[%{G}%H%? %1`%?%{g}][%= %{= kw}%-w%{+b yk} %n*%t%?(%u)%? %{-}%+w %=%{g}][%{B}%m/%d %{W}%C%A%{g}]'

	# huge scrollback buffer
	defscrollback 5000

	# no welcome message
	startup_message off

	Latency Comparison Numbers (~2012)
	----------------------------------
	L1 cache reference 0.5 ns
	Branch mispredict 5 ns
	L2 cache reference 7 ns 14x L1 cache
	Mutex lock/unlock 25 ns
	Main memory reference 100 ns 20x L2 cache, 200x L1 cache
	Compress 1K bytes with Zippy 3,000 ns 3 us
	Send 1K bytes over 1 Gbps network 10,000 ns 10 us
	Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD

	#!/bin/bash

	if [ $# -lt 2 ]; then
	echo "usage:$0 dev output_dir [iodepth]"
	echo "example 1: Testing the whole block device. Attention: That will destory the filesystem on the target block device"
	echo "./run_fio.sh /dev/sdb fio_test"
	echo ""
	echo "example 2: Testing a file, but not destory filesystem. Suppose the target device mount on /data"
	echo "fallocate -l 1G /data/test.dat"
	echo "./run_fio.sh /data/test.dat fio_test"

	AUDIT type=%{WORD:audit_type} msg=audit\(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}\): user pid=%{NUMBER:audit_pid} uid=%{NUMBER:audit_uid} auid=%{NUMBER:audit_audid} subj=%{WORD:audit_subject} msg=%{GREEDYDATA:audit_message}
	AUDITLOGIN type=%{WORD:audit_type} msg=audit\(%{NUMBER:audit_epoch}:%{NUMBER:audit_counter}\): login pid=%{NUMBER:audit_pid} uid=%{NUMBER:audit_uid} old auid=%{NUMBER:old_auid} new auid=%{NUMBER:new_auid} old ses=%{NUMBER:old_ses} new ses=%{NUMBER:new_ses}