Install_Kafka.md

I. Create a service User for Kafka:

As Kafka is a network application creating a non sudo user specifically for Kafka minimizes the risk if the machine is to be compromised.

sudo adduser --system --no-create-home --disabled-password --disabled-login kafka

II. Installing Kafka:

curl "http://www-eu.apache.org/dist/kafka/1.1.0/kafka_2.12-1.1.0.tgz" -o ~/Downloads/kafka.tgz

Create a directory for extracting Kafka

sudo mkdir /opt/kafka
sudo tar -xvzf ~/Downloads/kafka.tgz --directory /opt/kafka --strip-components 1

III. Configuring Zookeeper:

Create zookeeper data directory.

sudo mkdir /var/lib/zookeeper
sudo mkdir /var/lib/zookeeper/data

Ensure Permission of Directories:

sudo chown -R kafka:nogroup /var/lib/zookeeper

Config Zookeeper dataDir:

sudo nano /opt/kafka/config/zookeeper.properties

Change the directory dataDir

# the directory where the snapshot is stored.
dataDir=/var/lib/zookeeper/data

Create the unit file for zookeeper:

sudo nano /etc/systemd/system/zookeeper.service

Enter the following unit definition into the file:

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

IV. Configuring Kafka Server:

Kafka persists data to disk so we will now make a directory for it.

sudo mkdir /var/lib/kafka
sudo mkdir /var/lib/kafka/data

Ensure Permission of Directories:

sudo chown -R kafka:nogroup /opt/kafka
sudo chown -R kafka:nogroup /var/lib/kafka

Open /opt/kafka/config/server.properties, we will change Socket Server Settings and Log directory:

#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092

# A comma separated list of directories under which to store log files
log.dirs=/var/lib/kafka/data

Next, create the systemd service file for kafka:

sudo nano /etc/systemd/system/kafka.service

Enter the following unit definition into the file:

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

V. Start Server:

Now that the units have been defined, start Zookeeper and Kafka with the following command:

sudo service zookeeper start
sudo systemctl start kafka

To ensure that the server has started successfully, check the journal logs for the kafka unit:

sudo journalctl -u kafka