Kafka how to configure and debug when in need what to use to check.

Kafka_setup_debug

OS: Centos
Requisite: JAVA 7+8

sudo vi /etc/profile
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
export JRE_HOME=/usr/lib/jvm/jre

Download Kafka from any of the given website: the one use here is kafka_2.11-0.10.1.1.tgz
Untar:  tar -xvf kafka_2.11-0.10.1.1.tgz
sudo mv kafka_2.11-0.10.1.1.tgz /opt

To operate anything on Kafka go to its folder
cd /opt/kafka_2.11-0.10.1.1

Lets Configure: 

Requisites for kafka to setup and work: 
Zookeeper and Kafka broker setup

Start zookeeper 
$ bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

Configure Kafka server in kafka-server-start.sh 
Modify KAFKA_HEAP_OPTS = "-Xmx1G -Xms1G"  to higher if more memory is required


Kafka- server is ready to be started by specifying the brokers
Multiple Brokers can be created by having multiple server.properties. 

Start the kafka server -- which means to say have multiple brokers
$ bin/kafka-server-start.sh --daemon config/server.properties [--daemon to run as daemon process]
$ bin/kafka-server-start.sh --daemon config/server-1.properties
$ bin/kafka-server-start.sh --daemon config/server-2.properties

Create a topic called 'mytopic' and notify to the zookeeper

$ bin/kafka-topics.sh --create--zookeeper localhost:2181 --topic mytopic --replication-factor 3 --partitions 1  --config min.insync.replicas=3


Just check whether the topic is created by listing the topics
$ bin/kafka-topics.sh --list --zookeeper localhost:2181

Host a kafka consumer in  a separate terminal. 
$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mytopic --from-beginning

Ok ALL SET FOR KAFKA to have it run. 

#######################################
What if I have to debug or change config wrt Kafka / zookeeper

Lets  change the  min.insync.replicas  
$ bin/kafka-topics.sh --alter --zookeeper localhost:2181 --topic mytopic --replication-factor 3 -partitions 1  --config min.insync.replicas=2

Check whether the change is reflected
$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic mytopic


Kafka topic registered in the zookeeper is to be disabled or removed. 
Go to Zookeeper console which is more efficient. 

$ bin/zookeeper-shell.sh localhost:2181
>> opens a zookeeper console -shell
ls /brokers/topics
[mytopic]
rmr /brokers/topics/mytopic