Eucalyptus Logstash Filter

logstash.conf

input {
         tcp { port => 3333 }
 }
filter {
     grok {
         match => { "message" => ["%{TIMESTAMP_ISO8601:euca_timestamp}%{SPACE}%{LOGLEVEL:euca_log_level}%{SPACE}\|%{GREEDYDATA:euca_message}",
                                  "%{TIMESTAMP_ISO8601:euca_timestamp}%{SPACE}%{LOGLEVEL:euca_log_level}%{SPACE}%{BASE10NUM:euca_pid}%{SPACE}%{WORD:euca_function}%{SPACE}\|%{GREEDYDATA:euca_message}",
                                  "%{TIMESTAMP_ISO8601:euca_timestamp}%{SPACE}%{LOGLEVEL:euca_log_level}%{SPACE}%{GREEDYDATA:euca_message}"] }
     }
     date {
         match => ['euca_timestamp', "YYYY-MM-dd HH:mm:ss", "EEE MMM dd HH:mm:ss YYYY"]
     }
}
output { elasticsearch_http { host => "localhost" port => "9200" } }

In order to use this, first ensure you have Java 1.7 installed and download ELK:

java -version
wget https://download.elastic.co/elasticsearch/elasticsearch/elasticsearch-1.7.2.zip
unzip elasticsearch-1.7.2.zip
wget https://download.elastic.co/logstash/logstash/logstash-1.5.4.zip
unzip logstash-1.5.4.zip
wget https://download.elastic.co/kibana/kibana/kibana-4.1.2-darwin-x64.zip
unzip kibana-4.1.2-darwin-x64.zip

Start the ELK stack:

pushd elasticsearch-1.7.2; ./bin/elasticsearch & popd
pushd logstash-1.5.4; ./bin/logstash -f logstash.conf & popd # logstash.conf should contain whats in the gist
pushd kibana-4.1.2-darwin-x64; ./bin/kibana & popd

Load up your eucalyptus log files into Logstash. In this case I'm pulling them from my unzipped sosreport archives:

cat sosreport-*/var/log/eucalyptus/* | nc localhost 3333

Visit http://localhost:5601 to browse/search/splice your logs via Kibana.