kafka installation on ubuntu 16.04

Apache Kafka on Ubuntu.txt

Install and configure Apache Kafka on Ubuntu 16.04

# preparation
sudo apt-get update -y

1. Install Java (JDK8)::

Add the repository

  $ sudo add-apt-repository -y ppa:webupd8team/java

Update the metadata of the new repository and install JDK

  $ sudo apt update && apt install oracle-java8-installer -y

Verify that JDK 8 is installed properly

  $ java -version

You should see the output something like this

  java version "1.8.0_111"
  Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
  Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

2. Install zookeeper::

  $ sudo apt install zookeeperd
  
After the installation completes, ZooKeeper will be started as a daemon automatically. By default, it will listen on port 2181.

Confirm zookeeper is running on expected port::

  $ netstat -ant | grep :2181

If everything's fine, you should see the following output:

  tcp6       0      0 :::2181                 :::*                    LISTEN

if after typing 'ruok' once connected to 'localhost', zookeeper will respond with 'imok' and close the session.

  $ telnet localhost 2181
  Trying ::1...
  Connected to localhost.
  Escape character is '^]'.
  ruok <-- Type at empty prompt!
  imokConnection closed by foreign host.
    
3. Download kafka from http://kafka.apache.org/downloads.html::

  $ curl -O "http://www-eu.apache.org/dist/kafka/1.0.1/kafka_2.12-1.0.1.tgz"

Untar and move binaries to /opt/kafka/:

  $ sudo mkdir -p /opt/kafka && sudo tar -xvf kafka_2.11-0.10.1.1.tgz -C /opt/kafka/
  
4. Configure and run Kafka Server::

Turn on topic delete
  
  $ sudo sed -i 's/#delete.topic.enable=true/delete.topic.enable=true/' /opt/kafka/kafka_2.11-0.10.1.1/config/server.properties
  
Start Kafka by running kafka-server-start.sh script

  $ sudo /opt/kafka/kafka_2.11-0.10.1.1/bin/kafka-server-start.sh /opt/kafka/kafka_2.11-0.10.1.1/config/server.properties

Now, we can check listening ports:
- ZooKeeper : 2181
- Kafka 

  $ netstat -ant | grep -E ':2181|:9092'

If everything's fine, you should see the following output:

  tcp6       0      0 :::9092                 :::*                    LISTEN      1367/java       
  tcp6       0      0 :::2181                 :::*                    LISTEN      -               
  
5.  Test Server::

Open another session, and create a topic:

  $ /opt/kafka/kafka_2.11-0.10.1.1/bin/kafka-topics.sh --create --topic topic-test --zookeeper localhost:2181 --partitions 1 --replication-factor 1

List available topics:

  $ /opt/kafka/kafka_2.11-0.10.1.1/bin/kafka-topics.sh --list --zookeeper localhost:2181
    
Send message to topic as a producer via the 'kafka-console-producer.sh'::
  
    echo "hello world" | /opt/kafka/kafka_2.11-0.10.1.1/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic topic-test
    
*Consume* the send message::

   $ /opt/kafka/kafka_2.11-0.10.1.1/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-test --from-beginning

# referecing:
# https://gist.github.com/monkut/07cd1618102cbae8d587811654c92902
# https://devops.profitbricks.com/tutorials/install-and-configure-apache-kafka-on-ubuntu-1604-1/

configure-autostart.sh

cd /etc/init.d
sudo vi kafka
# paste the contents in etc-init.d-kafka
sudo chmod 755 kafka
sudo chmod 755 /etc/init.d/kafka
sudo update-rc.d kafka defaults

etc-init.d-kafka

DAEMON_PATH=//opt/kafka/kafka_2.12-1.0.1
PATH=$PATH:$DAEMON_PATH/bin

# See how we were called.
case "$1" in
  start)
        # Start daemon.
        echo "Starting Kafka";
        nohup $DAEMON_PATH/bin/kafka-server-start.sh -daemon /$DAEMON_PATH/config/server.properties 2> /dev/null
        ;;
  stop)
        echo "Shutting down Kafka";
        pid=`ps ax | grep -i 'kafka.Kafka' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          kill -9 $pid
        else
          echo "Kafka was not Running"
        fi
        ;;
  restart)
        $0 stop
        sleep 2
        $0 start
        ;;
  status)
        pid=`ps ax | grep -i 'kafka.Kafka' | grep -v grep | awk '{print $1}'`
        if [ -n "$pid" ]
          then
          echo "Kafka is Running as PID: $pid"
        else
          echo "Kafka is not Running"
        fi
        ;;
  *)
        echo "Usage: $0 {start|stop|restart|status}"
        exit 1
esac

exit 0

ref: https://stackoverflow.com/questions/34512287/how-to-automatically-start-kafka-upon-system-startup-in-ubuntu-14-04