kerberized_kafka.md

This is a tutorial on how to setup Kafka 0.9.0.0 on CentOS 7 with Kerberos. This configuration aims to set up all services on a single host called myserver.domain.com in the Kerberos realm DOMAIN.COM for testing purposes

Install JCE

This configuration requires that you have the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files installed. You can download these from the Java Downloads page under the Additional Resources section.

Install Kerberos KDC

Ashrithr made a very helpful tutorial on how to install Kerberos. Please use these instructions to set up Kerberos.

Prepare Kerberos principals

Create a principal and keytab for both kafka and zookeeper, using the following commands

[root@myserver ~]# mkdir -p /etc/security/keytabs
[root@myserver ~]# kadmin.local
kadmin.local: addprinc -randkey kafka/[email protected]
kadmin.local: ktadd -k /etc/security/keytabs/kafka.keytab kafka/[email protected]
kadmin.local: addprinc -randkey zookeeper/[email protected]
kadmin.local: ktadd -k /etc/security/keytabs/zookeeper.keytab zookeeper/[email protected]
kadmin.local: exit

Install Kafka

Download kafka_2.11-0.9.0.0.tgz, extract it to the /opt directory and create symlinks for convenience.

[root@myserver ~]# cd /tmp
[root@myserver tmp]# curl -O http://[apache mirror]/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz
[root@myserver opt]# cd /opt
[root@myserver opt]# tar -xvf /tmp/kafka_2.11-0.9.0.0.tgz
[root@myserver opt]# ln -s /opt/kafka_2.11-0.9.0.0 /opt/kafka
[root@myserver opt]# ln -s /opt/kafka/config /etc/kafka

Configure the Kafka Broker and Zookeeper

Add or change the following properties in the /etc/kafka/server.properties file for kafka.

listeners=PLAINTEXT://myserver.domain.com:9092,SASL_PLAINTEXT://myserver.domain.com:9093
sasl.kerberos.service.name=kafka
principal.to.local.class=kafka.security.auth.KerberosPrincipalToLocal
super.users=user:kafka
zookeeper.set.acl=false
zookeeper.connect=myserver.domain.com:2181
security.inter.broker.protocol=SASL_PLAINTEXT

Add or change the following properties in the /etc/kafka/zookeeper.properties file for zookeeper.

authProvider.0=org.apache.zookeeper.server.auth.SASLAuthenticationProvider

Create a JAAS server configuration file named /etc/kafka/kafka_jaas.conf with the following content for both kafka and zookeeper.

KafkaServer {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=false
        keyTab="/etc/security/keytabs/kafka.keytab"
        principal="kafka/[email protected]";
};
Client {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       storeKey=true
       useTicketCache=false
       keyTab="/etc/security/keytabs/kafka.keytab"
       principal="kafka/[email protected]";
};
Server {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       storeKey=true
       useTicketCache=false
       keyTab="/etc/security/keytabs/zookeeper.keytab"
       principal="zookeeper/[email protected]";
};    

Change the Kafka bash scripts to include the JAAS file

Edit the /opt/kafka/bin/kafka-run-class.sh to include the following:

#JAAS config file params
if [ -z "$KAFKA_JAAS" ]; then
  KAFKA_JAAS=""
fi

# Add the JAAS environment variable to the execution lines at the end of the script

# Launch mode
if [ "x$DAEMON_MODE" = "xtrue" ]; then
  nohup $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS $KAFKA_JAAS -cp $CLASSPATH $KAFKA_OPTS "$@" > "$CONSOLE_OUTPUT_FILE" 2>&1 < /dev/null &
else
  exec $JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS $KAFKA_JAAS -cp $CLASSPATH $KAFKA_OPTS "$@"
fi

Create the script /etc/profile.d/jaas.sh to set the JAAS environment variable

export KAFKA_JAAS="-Djava.security.auth.login.config=/etc/kafka/kafka_jaas.conf"

Run the jaas.sh script or start a new session.

Start the services

Start the Zookeeper and Kafka Broker

[root@myserver ~]# /opt/kafka/bin/zookeeper-server-start.sh -daemon /etc/kafka/zookeeper.properties
[root@myserver ~]# /opt/kafka/bin/kafka-server-start.sh -daemon /etc/kafka/server.properties

Leave the -daemon parameter out to run the services interactively.

Tips

• Disable firewalld in a test configuration to make your life easier.
• If you encounter a GSSException: Failure unspecified at GSS-API level (Mechanism level: Encryption type AES256CTS mode with HMAC SHA1-96 is not supported/enabled) on Zookeeper, it means that you did not install JCE.
• If you only use a SASL_PLAINTEXT listener on the Kafka Broker, you have to make sure that you have set the security.inter.broker.protocol=SASL_PLAINTEXT too, otherwise you will get a LEADER_NOT_AVAILABLE error in the client.

Related websites

• Kafka Security with Kerberos