Kafka useful commands

Kafka cheet sheets.md

Kafka

get size for all topics on worker

 for topic in $(find   /opt/cloudera/data/*/kafka/ -maxdepth 1 -mindepth 1 -type d -exec basename {} \;| awk -F '-' '{print $1}'| sort |uniq | grep -v consumer_offsets);do

du -csh /opt/cloudera/data/*/kafka/${topic}*| grep total| awk '{print "topic '$topic' "$1}'
done

get config for topic

 kafka-configs --describe --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka  --entity-type topics --entity-name  adrequests

get config adjustments for all topics

 kafka-configs --describe --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka  --entity-type topics

create topic

kafka-topics --create --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka --replication-factor 3 --partitions 50 evnt_v1_rtb

delete topic

kafka-topics --delete --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka --topic events_adreqs

change topic retention

# 1 day = 86400000, 7 day = 604800000
kafka-topics --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka --alter --topic evnt_v1_areq --config retention.ms=21600000

describe kafka topic

kafka-topics --describe --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka evnt_v1_opps

list kafka topics

kafka-topics --list --zookeeper hdp-m-01:2181/kafka

describe all kafka topics

kafka-topics --describe --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka

kafka-topics --describe --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka  --entity-type topics

kafka-topics --describe --zookeeper hdp-m-01:2181,hdp-m-02:2181,hdp-m-03:2181/kafka  --topics-with-overrides

consuming massages from the command line

kafka-console-consumer evnt_v1_imprs --zookeeper hdp-m-01:2181/kafka

find events with empty network_id for example

timeout 30 kafka-console-consumer evnt_v1_opps --zookeeper hdp-m-01:2181/kafka 2>&1 | grep hostname  | jq 'select(.network_id == null)'

rebalance cluster

https://www.cloudera.com/documentation/kafka/latest/topics/kafka_performance.html#concept_exp_hzk_br_unique_1

more info

https://archive.cloudera.com/kafka/kafka/2/kafka-0.9.0-kafka2.0.1/ops.html

retention policy

http://www.allprogrammingtutorials.com/tutorials/configuring-messages-retention-time-in-kafka.php