kafka install systemd.md

Kafka installation with systemd

0. Create kafka user

sudo adduser kafka
sudo adduser kafka sudo
su -l kafka

1. Download and Install kafka archive

cd /opt                                                              # go to /opt folder

curl  https://dlcdn.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz -O   # download archive to folder
tar -zxvf kafka*.tgz                                                 # extract archive 
rm kafka*.tgz                                                        # remove archive 
ln -s kafka_2.13-3.2.0/ kafka                                        # create symbolic link  

2. Create zookeeper service file replace {{ }} with the correct value in your case

# just creates a file with the following content  
cat <<EOF > /etc/systemd/system/kafka-zookeeper.service            

[Unit]
Description=Apache Zookeeper server (Kafka)
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User={{USER || kafka}}
Environment=JAVA_HOME={{must be JAVA_HOME here || /usr/lib/jvm/java-1.8.0-openjdk}}
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh

[Install]
WantedBy=multi-user.target

EOF

3. Create kafka service file

# just creates a file with the following content  
cat <<EOF > /etc/systemd/system/kafka.service

[Unit]
Description=Apache Kafka server (broker)
Documentation=http://kafka.apache.org/documentation.html
Requires=network.target remote-fs.target
After=network.target remote-fs.target kafka-zookeeper.service

[Service]
Type=simple
User={{USER || kafka}}
Environment=JAVA_HOME={{must be JAVA_HOME here || /usr/lib/jvm/java-1.8.0-openjdk}}
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh

[Install]
WantedBy=multi-user.target

EOF

3 * Edit service files

nano /etc/systemd/system/kafka-zookeeper.service
nano /etc/systemd/system/kafka.service

4. Edit kafka settings

nano /opt/kafka/config/server.properties
listeners=PLAINTEXT://{{IP || 192.168.0.103}}:9092

5. Reload and start the systemd services

systemctl daemon-reload
systemctl enable kafka-zookeeper.service
systemctl enable kafka.service
systemctl start kafka-zookeeper.service
systemctl start kafka.service

systemctl status kafka-zookeeper.service
systemctl status kafka.service

6. Check services state

systemctl status kafka-zookeeper.service
systemctl status kafka.service

very helpful, thank you so much!

Thanks :)

thank you

Thank you!!

Thank you :)

User=nano & Group=nano
There is no "nano" found.
Should I create login "nano" in /etc/passwd and group "nano" in /etc/group ?

User=nano & Group=nano
There is no "nano" found.
Should I create login "nano" in /etc/passwd and group "nano" in /etc/group ?

no
change it to your username
nano its just in my case

Thanks for posting this. I wonder how you add additional brokers to the service? Could you include an example of a service with 2 brokers?

the "http://apache-mirror.rbc.ru/pub/apache/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz" is not found

the "http://apache-mirror.rbc.ru/pub/apache/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz" is not found

you can choose the version available in the mirror by visiting this link: http://apache-mirror.rbc.ru/pub/apache/kafka/

thank you

thank you

Thx a lot!

is good, but if you are trying to setup a kafka broker in a remote machine were o only have a ssh conection to that machine, u should create another user and give him permissions to sudo, like explained in this guide:
https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-20-04

kafka.service - Apache Kafka Server
Loaded: loaded (/etc/systemd/system/kafka.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2021-06-03 16:57:10 IST; 20s ago
Docs: http://kafka.apache.org/documentation.html
Process: 5376 ExecStart=/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties (code=exited, status=1/FAILURE)
Main PID: 5376 (code=exited, status=1/FAILURE)

Jun 03 16:57:10 minal systemd[1]: Started Apache Kafka Server.
Jun 03 16:57:10 minal kafka-server-start.sh[5376]: Classpath is empty. Please build the project first e.g. by running './gradlew jar -PscalaVe
Jun 03 16:57:10 minal systemd[1]: kafka.service: Main process exited, code=exited, status=1/FAILURE
Jun 03 16:57:10 minal systemd[1]: kafka.service: Failed with result 'exit-code'.

server is not starting and I'm getting this response when I check kafka status
If you could help

thanks

Very helpful. Thank you

Thank you