As Kafka is a network application creating a non sudo user specifically for Kafka minimizes the risk if the machine is to be compromised.
sudo adduser --system --no-create-home --disabled-password --disabled-login kafka
curl "http://www-eu.apache.org/dist/kafka/1.1.0/kafka_2.12-1.1.0.tgz" -o ~/Downloads/kafka.tgz
Create a directory for extracting Kafka
sudo mkdir /opt/kafka
sudo tar -xvzf ~/Downloads/kafka.tgz --directory /opt/kafka --strip-components 1
Create zookeeper data directory.
sudo mkdir /var/lib/zookeeper
sudo mkdir /var/lib/zookeeper/data
Ensure Permission of Directories:
sudo chown -R kafka:nogroup /var/lib/zookeeper
Config Zookeeper dataDir:
sudo nano /opt/kafka/config/zookeeper.properties
Change the directory dataDir
# the directory where the snapshot is stored.
dataDir=/var/lib/zookeeper/data
Create the unit file for zookeeper:
sudo nano /etc/systemd/system/zookeeper.service
Enter the following unit definition into the file:
[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
Kafka persists data to disk so we will now make a directory for it.
sudo mkdir /var/lib/kafka
sudo mkdir /var/lib/kafka/data
Ensure Permission of Directories:
sudo chown -R kafka:nogroup /opt/kafka
sudo chown -R kafka:nogroup /var/lib/kafka
Open /opt/kafka/config/server.properties
, we will change Socket Server Settings and Log directory:
# listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://:9092
# A comma separated list of directories under which to store log files
log.dirs=/var/lib/kafka/data
Next, create the systemd service file for kafka:
sudo nano /etc/systemd/system/kafka.service
Enter the following unit definition into the file:
[Unit]
Requires=zookeeper.service
After=zookeeper.service
[Service]
Type=simple
User=kafka
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
Now that the units have been defined, start Zookeeper and Kafka with the following command:
sudo service zookeeper start
sudo systemctl start kafka
To ensure that the server has started successfully, check the journal logs for the kafka unit:
sudo journalctl -u kafka