Apache Kafka

Apache Kafka

=== Java install ===

java zookeeper-3.4.10.jar
sudo apt-add-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
sudo apt install oracle-java8-set-default
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
sudo update-alternatives --config java
sudo update-alternatives --config javac


=== Install Zookeeper ===

sudo adduser hadoop
sudo mkdir -p /data/zookeeper
sudo chown -R hadoop:hadoop /data/zookeeper

cd /opt
sudo wget http://www-eu.apache.org/dist/zookeeper/stable/zookeeper-3.4.10.tar.gz
sudo tar -xvf zookeeper-3.4.10.tar.gz
sudo chown hadoop:hadoop -R  zookeeper-3.4.10

sudo passwd kafka  ['kafka']

=== Install daemon ===
sudo apt-get install zookeeperd

=== Test zookeepr ===
telnet localhost 2181
ruok

=== Get Kafka ===
wget http://ftp.carnet.hr/misc/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz

=== Configure Kafka ===
vi config/server.properties

# Add in config/server.properties
delete.topic.enable = true

# Start Kafka
sh kafka-server-start.sh /opt/kafka/kafka_2.11-1.0.0/config/server.properties
# or in background:
nohup kafka-server-start.sh /opt/kafka/kafka_2.11-1.0.0/config/server.properties > ~/kafka/kafka.log 2>&1 &

# If started check: INFO [KafkaServer id=0] started

=== Test Kafka ===
# Create topic

bin/kafka-topics.sh --create \
    --zookeeper localhost:2181 \
    --replication-factor 1 \
    --partitions 1 \
    --topic TutorialTopic

# View topics
bin/kafka-topics.sh --list --zookeeper localhost:2181

# Set config
sed "116s/.*/zookeeper.connect=$ZK_PORT_2181_TCP_ADDR:$ZK_PORT_2181_TCP_PORT/" config/server.properties > config/server1.properties

# Start daemon
bin/kafka-server-start.sh -daemon config/server1.properties

# Create topic
bin/kafka-topics.sh --create --zookeeper $ZK_PORT_2181_TCP_ADDR:$ZK_PORT_2181_TCP_PORT --replication-factor 1 --partitions 1 --topic test

# Check topic
bin/kafka-topics.sh --describe --zookeeper $ZK_PORT_2181_TCP_ADDR:$ZK_PORT_2181_TCP_PORT --topic test

# Send mmessage
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test

$ Receive message
bin/kafka-console-consumer.sh --topic test --zookeeper $ZK_PORT_2181_TCP_ADDR:$ZK_PORT_2181_TCP_PORT --from-beginning

# Get status replications and partition
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test

# Get process
ps aux | grep server-1.properties

# Configuration
compression.type = gzip,snappy,lz4 | Cool bench: https://github.com/lz4/lz4
------

=== Ecosystem ===
Kafka Manager: https://github.com/yahoo/kafka-manager

=== Kafka Manager ===
wget https://github.com/yahoo/kafka-manager/archive/master.zip
unzip master.zip
mv kafka-manager-master kafka-manager

sudo apt-get install scala

sudo apt-get install default-jdk

wget -c https://bintray.com/artifact/download/sbt/debian/sbt-0.13.9.deb

Go to Kafka Manager directory:
cd kafka-manager
sudo sbt clean dist

sudo mv target/universal/kafka-manager-1.3.3.15.zip ~/
cd ~ && unzip kafka-manager-1.3.3.15.zip

# Run
bin/kafka-manager -Dkafka-manager.zkhosts="localhost:2181"
http://localhost:9000
# Create Zookeepr 127.0.0.1 with cluster name

=== Kafka tool ===
# GUI view messages
http://www.kafkatool.com/download.html
sh kafkatool.sh
then install via GUI.

=== Realtime disk I/O
sudo apt install iotop

=== Kafka guides ===
# Stop service Kafka
sh kafka-server-start.sh /opt/kafka/kafka_2.11-1.0.0/config/server.properties

# Kafka systemd
https://gist.github.com/atrepca/c3e811d6cc6529992a7281d6dcdd54be

# Test perf
bin/kafka-consumer-perf-test.sh --topic benchmark-3-3-none --zookeeper localhost:2181 --messages 15000000 --threads 1

=== Requirements ===
* any Linux/Solaris
* ext4, xfs
* Test: kafka-producer-perf-test.sh

=== Calc ===
Consumption: 17000 msg/s
Size of message: 600 bytes

RESULT:

Throughput 
(17000 * 600 / (1024*1024)) = 9.73 MB/s

5 brokers (+ 3 replications)
9.73 MB/s / 5*3 = 0.64 = ~6 MB/s

30 seconds buffer = 6 MB/s * 30sec = 180 MB memory

=== Quick tip ====

Kafka manager:
cd /opt/kafkamanager/kafka-manager/target/universal/kafka-manager-1.3.3.16
bin/kafka-manager -Dkafka-manager.zkhosts="localhost:2181"

=== Fix to view Metrics on Kafka Manager ===
I recommended Method 2.

Method 1 (hard way):

Make sure that is KAFKA_JMX_OPTS and KAFKA_JMX_PORT values like:

export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=10000 -Djava.rmi.server.hostname=$HOSTNAME"
export KAFKA_JMX_PORT=9092
export JMX_PORT=9999
/opt/kafka/kafka_2.11-1.0.0$ bin/kafka-server-stop.sh 

bin/kafka-server-start.sh -daemon config/server1.properties

Start messaging brokers

Then restart kafka-manager:
Kill kafka-manager
bin/kafka-manager -Dkafka-manager.zkhosts="localhost:2181"

Now you can see Metrics successfuly.

Method 2 (easy way):
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=10000 -Djava.rmi.server.hostname=$HOSTNAME"
Add export JMX_PORT=${JMX_PORT:-9999} to $KAFKA_HOME/bin/kafka-server-start.sh before the line: exec $base_dir/kafka-run-class.sh.

Start Kafka Manager