Instructions to test Logstash with Kafka

indexer.conf

input {
  kafka {
    host => "127.0.0.1"
  port => "9092"
    type => "redis-input"
    topic => "logstash"
    message_format => "json_event"
  }
}

output {
  stdout { debug => true debug_format => "json"}

  elasticsearch {
    host => "127.0.0.1"
  }
}

kafka.conf

input {
  stdin {
    type => "stdin-type"
  }
}

output {
  stdout { debug => true debug_format => "json"}
  kafka { host => "127.0.0.1" port => 9092 topic => "logstash" }
}

Readm.md

Test Logstash with Kafka

Setup Kafka

Download Kafka from:

https://www.apache.org/dyn/closer.cgi/incubator/kafka/kafka-0.7.2-incubating/kafka-0.7.2-incubating-src.tgz

Install Kafka

tar xzf kafka-0.7.2-incubating-src.tgz
cd kafka-0.7.2-incubating-src
./sbt update
./sbt package

Start Zookeeper server

bin/zookeeper-server-start.sh config/zookeeper.properties

Start Kafka server

bin/kafka-server-start.sh config/server.properties

Logstash

Setup logstash fork with Kafka input and outputs:

git clone [email protected]:jeroenvandijk/logstash.git
git checkout feature/kafka

# Install JRuby with rvm
rvm install jruby-1.7.1
rvm use jruby-1.7.1

# Install logstash dependencies
ruby gembag.rb logstash.gemspec

Open terminal 1)

bin/logstash agent -f kafka.conf -- web --backend 'elasticsearch:///?local'

And open terminal 2)

bin/logstash agent -f indexer.conf 

Type in terminal 1) and see output in terminal 2) :-)

I had some trouble getting this to work w/ the elasticsearch_http plugin as it in this repo. But I brought it up to date w/ the current tip of master, and got it working. Code here:
https://github.com/squito/logstash/tree/feature/kafka_0.7

and some discussion here:
https://groups.google.com/forum/#!topic/logstash-users/l1WkJVI6wFU

(note -- I'm not an expert on logstash, nor a ruby programmer, so maybe it worked fine in the beginning and I did something wrong, or maybe there are problems w/ the code I added)