weijh imweijh

logstash needs full time love.

There are roughly 70 code contributors to the logstash project as of today. Many more exist as helpful folks on IRC and the mailling list. Others post awesome bug reports and feature requests. The community is simply awesome.

I've been working on logstash for about 2 years. All as a hobby - all part-time. About 70% of the current content (commits, 'git blame' lines, etc) are by me.

Unfortunately, I've never used logstash in production. (Embarrassing, I know!)

I'm embarrassed to answer user questions like "How do I optimize elasticsearch's storage?" with "Sorry, I've never used elasticsearch outside of my laptop."

Long story, short: I'm totally open to supporting more rubies if possible. Details follow.

Related issue: http://code.google.com/p/logstash/issues/detail?id=37

Summary:

core and stdlib ruby changes violently and without notice and without backwards compatibility. I want nothing of that.
need a cross-ruby date library that isn't part of stdlib (see previous point) and is also good.
need an easy way to use multiple cpus that is cross-ruby (threads are not it)

	# the following two lines give a two-line status, with the current window highlighted
	hardstatus alwayslastline
	hardstatus string '%{= kG}[%{G}%H%? %1`%?%{g}][%= %{= kw}%-w%{+b yk} %n*%t%?(%u)%? %{-}%+w %=%{g}][%{B}%m/%d %{W}%C%A%{g}]'

	# huge scrollback buffer
	defscrollback 5000

	# no welcome message
	startup_message off

	设置防火墙，打标签，让fprobe-ulog记录（只处理对外的网卡eth0）
	iptables -I INPUT -i eth0 -j ULOG --ulog-cprange 48 --ulog-qthreshold 50
	iptables -I OUTPUT -o eth0 -j ULOG --ulog-cprange 48 --ulog-qthreshold 50
	iptables -I FORWARD -j ULOG --ulog-cprange 48 --ulog-qthreshold 50

	发送给 172.18.20.160:9995
	fprobe-ulog -B4096 -q10000 172.18.20.160:9995

	接收：
	nfcapd -z -w -D -T all -l /var/cache/nfdump -I any -S 2 -P /var/run/nfcapd.allflows.pid

	#!/bin/bash

	# Make sure we have the dev tools
	yum groupinstall "Development Tools"

	# Just in case you started installing dependencies from yum
	yum -y remove libnfnetlink

	# lets put the source code here
	mkdir -p ~/.src

	yum remove libnfnetlink libnfnetlink-devel

	mkdir ./src
	cd ./src

	wget http://netfilter.org/projects/libnfnetlink/files/libnfnetlink-1.0.1.tar.bz2
	tar xjvf libnfnetlink-1.0.1.tar.bz2
	cd libnfnetlink-1.0.1
	./configure && make && make install
	cd ..

	#################
	# Sflow Filters #
	#################

	filter {

	if [type] == "sflow" {

	# sFlow sends two kinds of messages - CNTRs and FLOWs
	# I'm not doing anything with CNTRs at this point, so

	# 用kibana展示metric.count最新值的方法：每天凌晨删除相关index，并重启logstash清空metrics事件，然后kibana取该字段的最大值...

	/bin/sed -i -r "s/$(date '+%Y%m%d' -d "1 day ago")/$(date '+%Y%m%d')/g" /etc/logstash/conf.d/logstash2.conf
	/usr/bin/curl --noproxy "" -XDELETE localhost:9200/metric-

	top -c -p $(pgrep "java\|samplicate" -d ",")
	# 只展示java或者samplicate命名的进程

	"0": HOPOPT
	"1": ICMP
	"2": IGMP
	"3": GGP
	"4": IPv4
	"5": ST
	"6": TCP
	"7": CBT
	"8": EGP
	"9": IGP