Install_Kafka_Multiple_Brokers.md

Step 1

Download the latest version of Apache Kafka. Extract the folder and move it to the bin folder. Start ZooKeeper, which is essential to start with the Kafka cluster. ZooKeeper is the coordination service to manage the brokers, leader election for partitions, and alerts when Kafka changes topics (i.e. deletes topic, creates topic, etc.) or brokers (add broker, dead broker, etc.). In this example, I have started only one ZooKeeper instance. In production environments, we should have more ZooKeeper instances to manage fail-over. Without ZooKeeper, the Kafka cluster cannot work.

Create a directory for extracting Kafka

sudo mkdir /opt/kafka
sudo tar -xvzf ~/Downloads/kafka.tgz --directory /opt/kafka --strip-components 1

./zookeeper-server-start.sh ../config/zookeeper.properties

Step 2

Now, start the Kafka brokers. In this example, we are going to start three brokers. Go to the config folder under Kafka root, copy the server.properties file three times, and name them server_1.properties, server_2.properties and server_3.properties. Change the below properties in those files.

#####server_1.properties#####
broker.id=1
listeners=PLAINTEXT://:9091
log.dirs=/tmp/kafka-logs-1
#####server_2.properties######
broker.id=2
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs-2
######server_3.properties#####
broker.id=3
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs-3

In /opt/kafka/ directory, start three brokers with the below commands.

###Start Broker 1 #######
./kafka-server-start.sh ../config/server_1.properties
###Start Broker 2 #######
./kafka-server-start.sh ../config/server_2.properties
###Start Broker 3 #######
./kafka-server-start.sh ../config/server_3.properties

Step 3

Create a topic with the below command.

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 3 --topic first_topic

Step 4

Produce some messages on the topic created in the above step using the Kafka console producer. For the console producer, mention any one of the broker addresses. That will be the bootstrap server to gain access to the entire cluster.

./kafka-console-producer.sh --broker-list localhost:9091 --topic first_topic
>First message
>Second message
>Third message
>Fourth message
>

Step 5

Consume the messages using the Kafka console consumer. For the Kafka consumer, mention any of the broker addresses as the bootstrap server. Remember that while reading the messages, you may not see the order, as the order is maintained at the partition level, not at the topic level.

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic first_topic --from-beginning

If you want, you can describe the topic to see how partitions are distributed and see the leaders of each partition using the below command.

./kafka-topics.sh --describe --zookeeper localhost:2181 --topic first_topic
#### The Result for the above command#####
Topic:first_topic	PartitionCount:3	ReplicationFactor:3	Configs:
	Topic: first_topic	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,2,3
	Topic: first_topic	Partition: 1	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1
	Topic: first_topic	Partition: 2	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2

In the above description, broker-1 is the leader for partition:0 and broker-1, broker-2, and broker-3 has replicas of each partition.