mtls-kafka.md

Kafka and mTLS Walkthrough

Creating a Certificate Authority (CA)

Build the Certificate

openssl req -new -newkey rsa:4096 -days 365 -x509 -subj "/CN=Kafka-Security-CA" -keyout ca-key -out ca-cert -nodes

You should see:

Generating a RSA private key
...................................................++++
..............................................................................................................................................................++++
writing new private key to 'ca-key'

And the directory will now contain

-rw-rw-r-- 1 ubuntu ubuntu 1830 Apr  7 21:58 ca-cert
-rw------- 1 ubuntu ubuntu 3272 Apr  7 21:58 ca-key

ca-key is the private key:

-----BEGIN PRIVATE KEY-----

ca-cert is the public certificate that can be imported into the trust store later:

-----BEGIN CERTIFICATE-----

You can now use this CA when you secure your Kafka broker.

Create the Keystore

keytool -genkey -keystore kafka.server.keystore.jks -validity 365 -storepass confluent -keypass confluent -dname "CN=host-dns-name.compute.amazonaws.com" -storetype pkcs12 -keyalg RSA

Where -dname is the public DNS name of the host

You will now see:

-rw-rw-r-- 1 ubuntu ubuntu 1830 Apr  7 21:58 ca-cert
-rw------- 1 ubuntu ubuntu 3272 Apr  7 21:58 ca-key
-rw-rw-r-- 1 ubuntu ubuntu 2359 Apr  7 22:22 kafka.server.keystore.jks

Check the jks file:

keytool -list -v -keystore kafka.server.keystore.jks

You'll see something like:

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Look at the Owner and make sure the CN matches:

Owner: CN=host-dns-name.compute.amazonaws.com

Create the cert file (a signed version of the certificate)

This step creates an output file called cert-file which the brokers can use. This step is the first part of a 2-step process and is commonly called a signing request.

keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass confluent -keypass confluent

You should now see a cert-file has been generated in the directory:

-rw-rw-r-- 1 ubuntu ubuntu 1509 Apr  7 22:42 cert-file

The cert-file that has just been output would normally be sent over to the CA Administrator to get the certificate signed. In our case, we're just going to self-sign the certificate.

Self Sign the Certificate

The most important switches are:

• req : a request
• CA : the Certificate Authority
• CAKey : the public key
• in : the input (the signing request)
• out : the signed certificate

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:confluent

You should see:

Signature ok
subject=CN = host-dns-name.compute.amazonaws.com
Getting CA Private Key

And your directory should now contain a cert-signed file:

-rw-rw-r-- 1 ubuntu ubuntu 2139 Apr  8 07:53 cert-signed

Examine the signed certificate file by running:

keytool -printcert -v -file cert-signed

Where:

• printcert : request to print the certificate
• v : uses verbose mode

You should see something like this in the output:

Owner: CN=host-dns-name.compute.amazonaws.com
Issuer: CN=Kafka-Security-CA
Serial number: 4a3dfd85781b74852c77194b7665b0b31870c75e
Valid from: Fri Apr 08 07:53:18 UTC 2022 until: Sat Apr 08 07:53:18 UTC 2023
Certificate fingerprints:
	 SHA1: 73:87:AC:60:B7:2F:6E:14:A8:01:D6:0C:63:5D:59:CB:B9:B1:B4:AB
	 SHA256: A2:C9:A2:A3:1A:EA:FC:CF:54:B4:84:1A:11:06:73:29:26:18:26:CF:42:1A:3D:42:2C:18:1B:5E:C9:81:C5:5D
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit DSA key
Version: 1

Note the Owner and the Issuer fields in the output.

Create the Trust Store

The next part of the process is to create our truststore on the Kafka Broker:

keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt -keyalg RSA

• keystore : the jks truststore
• alias : ???
• import : import our own CA public certificate

You should see:

Certificate was added to keystore

and in your directory listing, you should now see a new file (kafka.server.truststore.jks):

-rw-rw-r-- 1 ubuntu ubuntu 1671 Apr  8 09:30 kafka.server.truststore.jks

Import the certificate into the trust store

For the next stage, we need to import the new certificates into our keystore. Now we have a signed certificate and a ca-certificate; both of these need to be stored in the keystore.

First import the public ca-cert certificate into the keystore:

keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt

At this stage, all the switches should look familiar...

You should see the message:

Certificate was added to keystore

Then import the cert-signed signed certificate into the keystore:

keytool -keystore kafka.server.keystore.jks -import -file cert-signed -storepass confluent -keypass confluent -noprompt

You should see:

Certificate reply was installed in keystore

Configure the broker

vim kafka/config/server.properties

You'll be adding the following lines:

# TLS / SSL
ssl.keystore.location=/home/ubuntu/ssl/kafka.server.keystore.jks
ssl.keystore.password=confluent
ssl.key.password=confluent
ssl.truststore.location=/home/ubuntu/ssl/kafka.server.truststore.jks
ssl.truststore.password=confluent

You also need to ensure you have both listeners registered (the PLAINTEXT listener on port 9092 and the SSL listener on port 9093). You need to configure the listeners property in order for this mapping to be created in the broker.

You also need to extend advertised.listeners to add the new SSL listener; note that we're specifying the SSL port for your SSL connection in the advertised.listeners property:

listeners=PLAINTEXT://:9092,SSL://:9093
advertised.listeners=PLAINTEXT://host-dns-name.compute.amazonaws.com:9092,SSL://host-dns-name.compute.amazonaws.com:9093

Now we need to restart Kafka:

sudo systemctl restart kafka && sudo systemctl status kafka

We want to see that the broker is in a running state after the configuration changes have been made:

● kafka.service - Apache Kafka server (broker)
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-04-08 12:52:32 UTC; 14ms ago

Verify connectivity by running:

openssl s_client -connect host-dns-name.compute.amazonaws.com:9093 -cipher ALL

You should now see something like:

CONNECTED(00000003)
depth=1 CN = Kafka-Security-CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 CN = Kafka-Security-CA
verify return:1
depth=0 CN = host-dns-name.compute.amazonaws.com
verify return:1
---

Grep for:

grep "endpoint" kafka/logs/server.log

You should see the SSL listener reported as "Started":

[2022-04-09 08:41:37,820] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane acceptor and processor(s) for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer)
[2022-04-09 08:41:37,849] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane acceptor and processor(s) for endpoint : ListenerName(SSL) (kafka.network.SocketServer)

Check the key

keytool -keystore ssl/kafka.client.keystore.jks -list -v

Note that Criticality=false in some cases. BasicConstraints CA:true (self signed)

See: https://github.com/confluentinc/cp-demo/blob/7.1.0-post/scripts/security/certs-create-per-user.sh#L43

certs-create-per-user.sh

Putting it all together

All the commands back-to-back:

keytool -genkey -keystore kafka.server.keystore.jks -validity 365 -storepass confluent -keypass confluent -dname "CN=host-dns-name.compute.amazonaws.com" -storetype pkcs12 -keyalg RSA
keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass confluent -keypass confluent
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:confluent
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt -keyalg RSA
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt
keytool -keystore kafka.server.keystore.jks -import -file cert-signed -storepass confluent -keypass confluent -noprompt
sudo systemctl restart kafka && sudo systemctl status kafka

debugging notes

• openssl s_client -connect host-dns-name.compute.amazonaws.com:9093 -CAfile ssl/ca-cert -tls1_2
• openssl s_client -debug -connect host-dns-name.compute.amazonaws.com:9093

Securing the client

Prerequisites

First ensure that you have the ca-cert file (the public CA certificate) on your machine - in our case, we created this certificate on our remote Kafka broker machine, so we will need to scp it over:

scp -i ~/key.pem [email protected]:~/ssl/ca-cert .

Check the directory on the machine that will be used for the client application for the ca-cert file:

-rw-r--r--  1 ableasdale  staff   1.8K 10 Apr 09:01 ca-cert

Create the Client Trust Store

• ca-cert is the public CA certificate
• alias is the CA Certificate alias name - we're using the same CARoot alias that was used previously

This will create your truststore file

keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt -keyalg RSA

You should see the message:

Certificate was added to keystore

Check the keystore file:

keytool -list -v -keystore kafka.client.truststore.jks

You will be prompted for the password and then you should see something like this:

Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

Note the Alias name and the Owner:

Alias name: caroot
[...]
Owner: CN=Kafka-Security-CA
Issuer: CN=Kafka-Security-CA

Create the Client Certificate

This will create your keystore file (kafka.client.keystore.jks):

keytool -genkey -keystore kafka.client.keystore.jks -validity 365 -storepass confluent -keypass confluent -dname "CN=mylaptop" -alias my-local-pc -storetype pkcs12 -keyalg RSA

Create Certificate Signing Request

This will create the client-cert-sign-request file for your client application:

keytool -keystore kafka.client.keystore.jks -certreq -file client-cert-sign-request -alias my-local-pc -storepass confluent -keypass confluent

In a production situation, this signing request would be sent to your Certificate Admin for signing. However, as before, we're going to rely on self-signing.

SCP the file to our broker machine

scp -i ~/key.pem client-cert-sign-request [email protected]:~/ssl

You should now see the client-cert-sign-request in the ssl directory on the broker:

-rw-r--r-- 1 ubuntu ubuntu  977 Apr 10 07:19 client-cert-sign-request

SSH into our EC2 instance

ssh -i key.pem [email protected]

cd to the ssl directory

Sign the Certificate

Remember:

• ca-cert is the public CA certificate
• ca-key is the private key for the CA certificate
• in is the input file (client-certificate-sign-request)
• out is the output file (client-cert-signed)

As with the broker section, we will use the openssl commandline tool to sign the client certificate:

openssl x509 -req -CA ca-cert -CAkey ca-key -in client-cert-sign-request -out client-cert-signed -days 365 -CAcreateserial -passin pass:confluent

You should see:

Signature ok
subject=CN = mylaptop
Getting CA Private Key

Your directory listing will now contain the new client-cert-signed certificate:

-rw-rw-r-- 1 ubuntu ubuntu 1346 Apr 10 07:29 client-cert-signed

Copy the signed client certificate back to the client machine

We're going to use scp to copy the client certificate back to the machine running the client applications (in this case, it's a laptop):

From your client machine (i.e. your laptop), run:

scp -i ~/key.pem [email protected]:~/ssl/client-cert-signed .

You'll now see this in the directory:

-rw-r--r--  1 ableasdale  staff   1.3K 10 Apr 08:55 client-cert-signed

Import the signed client certificate and the CA into the keystore

Now you can add the public CA certificate into the client keystore. We SCPd this file over earlier in the process.

Notes:

• keystore is the client keystore file (we're adding the root certificate to the store)
• alias this is the CA Certificate so we're using the same CARoot alias that was used previously
• file is the public CA certificate

keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt

You should see:

Certificate was added to keystore

Now we add the client signed certificate (client-cert-signed) using the keytool command:

keytool -keystore kafka.client.keystore.jks -import -file client-cert-signed -alias my-local-pc -storepass confluent -keypass confluent -noprompt

You should see:

Certificate reply was installed in keystore

Configure the Kafka Broker to require SSL client Authentication

Now we need to edit the server.properties on the broker:

vim ~/kafka/config/server.properties

We need to add the following line to the broker's properties file:

# Client Authentication
ssl.client.auth=required

Restart Kafka to apply the new configuration changes:

sudo systemctl restart kafka && sudo systemctl status kafka

Create the Client SSL Authentication Properties File

On your client machine, you need to create a properties file:

vim client-ssl-auth.properties

We're adding the following lines to the client-ssl-auth.properties properties file:

security.protocol=SSL
ssl.truststore.location=~/ssl/kafka.client.truststore.jks
ssl.truststore.password=confluent
ssl.keystore.location=~/ssl/kafka.client.keystore.jks
ssl.keystore.password=confluent
ssl.key.password=confluent

Test the Producer

We're using kafka-console-producer.sh in the bin directory of Apache Kafka and we're passing in the client-ssl-auth.properties that we created in the previous step:

~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-producer.sh --broker-list host-dns-name.compute.amazonaws.com:9093 --topic kafka-security-topic --producer.config client-ssl-auth.properties

Test the Consumer

We're using kafka-console-consumer.sh in the bin directory of Apache Kafka and we're passing in the client-ssl-auth.properties that we created in the previous step:

~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server host-dns-name.compute.amazonaws.com:9093 --topic kafka-security-topic --consumer.config client-ssl-auth.properties

Non-TLS Producer and Consumer

Note that we have configured our broker with two listeners; there's the plaintext listener (on port 9092) and the SSL listener (on port 9093). We can still use the non-TLS endpoint to communicate with the brokers:

Producer

~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-producer.sh --broker-list host-dns-name.compute.amazonaws.com:9092 --topic kafka-security-topic

Consumer

~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server host-dns-name.compute.amazonaws.com:9092 --topic kafka-security-topic