Skip to content

Instantly share code, notes, and snippets.

@nsabharwal
Created April 29, 2015 06:12
Show Gist options
  • Save nsabharwal/600bef5a0454e0738a93 to your computer and use it in GitHub Desktop.
Save nsabharwal/600bef5a0454e0738a93 to your computer and use it in GitHub Desktop.
Indigesting to kafka by flume syslog
yum -y install sys-ng
# /etc/flume/conf/flume.conf
agent.sources=syslogsource-1
agent.channels=mem-channel-1
agent.sinks=kafka-sink-1
agent.sources.syslogsource-1.type=syslogtcp
agent.sources.syslogsource-1.port=13073
agent.sources.syslogsource-1.host=0.0.0.0
agent.sources.syslogsource-1.channels=mem-channel-1
agent.channels.mem-channel-1.type = memory
agent.sinks.kafka-sink-1.channel = mem-channel-1
agent.sinks.kafka-sink-1.type = org.apache.flume.sink.kafka.KafkaSink
agent.sinks.kafka-sink-1.batchSize = 5
agent.sinks.kafka-sink-1.brokerList = kafka_host:6667
agent.sinks.kafka-sink-1.topic = flume.auths
cd /usr/hdp/current/kafka-broker/bin
#create kafka topic
./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor=1 --partitions 1 --topic flume.auths
# watch data flow in via kafka cosole consumer
/kafka-console-consumer.sh --zookeeper localhost:2181 --topic flume.auths --from-beginning
#start flume agent
flume-ng agent --conf-file /etc/flume/conf/flume.conf --name agent
#create log messages
loggen -I 113 -s 113 -r 10 flumeagen-node 13073
@randerzander
Copy link

Well, I forked up. Thought I could make edits and send you a pull request :|

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment