Kafka console consumer with Kerberos

Kafka-console-consumer-with-kerberos.md

Kafka console consumer with Kerberos

1. Create a jaas.conf file with the following contents:

KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="keytabFile.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="kafka"
   debug=true
   principal="[email protected]";
};
Client {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="keytabFile.keytab"
   storeKey=true
   useTicketCache=false
   debug=true
   principal="[email protected]";
};

2. Create a properties file (say "consumer.properties") with the following contents:

security.protocol=SASL_PLAINTEXT
sasl.kerberos.service.name=kafka

3. Then at the terminal run the following command:

export KAFKA_OPTS="-Djava.security.auth.login.config=./jaas.conf"

4. List topics

kafka-topics.sh --list --zookeeper localhost:2181

5. Execute the Kafka-console-consumer script:

kafka-console-consumer --topic <topic-name> --from-beginning --bootstrap-server <anybroker>:9092 --consumer.config ./consumer.properties

6. Execute the Kafka-console-producer script:

kafka-console-producer --broker-list <anybroker>:9093 --topic topic-kumar --producer.config consumer.properties < event1.json 

7. Execute the Kafka-topic script to describe a given topic:

kafka-topics --describe --topic topic-kumar --zookeeper <zookeeperHost>:2181

8. Delete topic (provided delete.topic.enable is set)

./bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic remove-me

If the above doesn't work, try the following:

# Login to zookeeper and -
hbase zkcli
rmr /brokers/topics/<topic_name>
rmr /admin/delete_topics/<topic_name>