Kafka Distributed

* Install Java8

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

** Install SBT

echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install sbt


1. Start 3 Machine:

2. download Kafka Zip file:

3. Edit following :

start ZooKeeper:
----------------------------
$ bin/zookeeper-server-start.sh config/zookeeper.properties


Edit :

config/server.properties 


On Machine 1:

cp config/server.properties config/server1.properties

broker.id=1
port=9092
log.dir=/tmp/kafka-logs-1



On Machine 2:

cp config/server.properties config/server2.properties

broker.id=2
port=9093
log.dir=/tmp/kafka-logs-2


On Machine 3:
cp config/server.properties config/server3.properties

broker.id=3
port=9094
log.dir=/tmp/kafka-logs-3



Start Broker :

env JMX_PORT=9999  bin/kafka-server-start.sh config/server1.properties

env JMX_PORT=10000 bin/kafka-server-start.sh config/server2.properties

env JMX_PORT=10001 bin/kafka-server-start.sh config/server3.properties


Create Kafka Topic:

bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic Kryptos --partitions 3 --replication-factor 2


Discribe Topic:
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic


yum --disablerepo=epel -y update  ca-certificates

yum install -y python-pip

yum groupinstall -y 'development tools'

yum install libxml2

yum install libxslt-devel

sudo yum install -y gcc ruby-devel libxml2 libxml2-devel libxslt libxslt-devel


sudo yum install python-devel