openssl req -new -newkey rsa:4096 -days 365 -x509 -subj "/CN=Kafka-Security-CA" -keyout ca-key -out ca-cert -nodes
You should see:
Generating a RSA private key
...................................................++++
..............................................................................................................................................................++++
writing new private key to 'ca-key'
And the directory will now contain
-rw-rw-r-- 1 ubuntu ubuntu 1830 Apr 7 21:58 ca-cert
-rw------- 1 ubuntu ubuntu 3272 Apr 7 21:58 ca-key
ca-key
is the private key:
-----BEGIN PRIVATE KEY-----
ca-cert
is the public certificate that can be imported into the trust store later:
-----BEGIN CERTIFICATE-----
You can now use this CA when you secure your Kafka broker.
keytool -genkey -keystore kafka.server.keystore.jks -validity 365 -storepass confluent -keypass confluent -dname "CN=host-dns-name.compute.amazonaws.com" -storetype pkcs12 -keyalg RSA
Where -dname
is the public DNS name of the host
You will now see:
-rw-rw-r-- 1 ubuntu ubuntu 1830 Apr 7 21:58 ca-cert
-rw------- 1 ubuntu ubuntu 3272 Apr 7 21:58 ca-key
-rw-rw-r-- 1 ubuntu ubuntu 2359 Apr 7 22:22 kafka.server.keystore.jks
Check the jks file:
keytool -list -v -keystore kafka.server.keystore.jks
You'll see something like:
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
Look at the Owner and make sure the CN matches:
Owner: CN=host-dns-name.compute.amazonaws.com
This step creates an output file called cert-file
which the brokers can use. This step is the first part of a 2-step process and is commonly called a signing request.
keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass confluent -keypass confluent
You should now see a cert-file
has been generated in the directory:
-rw-rw-r-- 1 ubuntu ubuntu 1509 Apr 7 22:42 cert-file
The cert-file
that has just been output would normally be sent over to the CA Administrator to get the certificate signed. In our case, we're just going to self-sign the certificate.
The most important switches are:
- req : a request
- CA : the Certificate Authority
- CAKey : the public key
- in : the input (the signing request)
- out : the signed certificate
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:confluent
You should see:
Signature ok
subject=CN = host-dns-name.compute.amazonaws.com
Getting CA Private Key
And your directory should now contain a cert-signed
file:
-rw-rw-r-- 1 ubuntu ubuntu 2139 Apr 8 07:53 cert-signed
Examine the signed certificate file by running:
keytool -printcert -v -file cert-signed
Where:
- printcert : request to print the certificate
- v : uses verbose mode
You should see something like this in the output:
Owner: CN=host-dns-name.compute.amazonaws.com
Issuer: CN=Kafka-Security-CA
Serial number: 4a3dfd85781b74852c77194b7665b0b31870c75e
Valid from: Fri Apr 08 07:53:18 UTC 2022 until: Sat Apr 08 07:53:18 UTC 2023
Certificate fingerprints:
SHA1: 73:87:AC:60:B7:2F:6E:14:A8:01:D6:0C:63:5D:59:CB:B9:B1:B4:AB
SHA256: A2:C9:A2:A3:1A:EA:FC:CF:54:B4:84:1A:11:06:73:29:26:18:26:CF:42:1A:3D:42:2C:18:1B:5E:C9:81:C5:5D
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit DSA key
Version: 1
Note the Owner and the Issuer fields in the output.
The next part of the process is to create our truststore on the Kafka Broker:
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt -keyalg RSA
- keystore : the jks truststore
- alias : ???
- import : import our own CA public certificate
You should see:
Certificate was added to keystore
and in your directory listing, you should now see a new file (kafka.server.truststore.jks
):
-rw-rw-r-- 1 ubuntu ubuntu 1671 Apr 8 09:30 kafka.server.truststore.jks
For the next stage, we need to import the new certificates into our keystore. Now we have a signed certificate and a ca-certificate; both of these need to be stored in the keystore.
First import the public ca-cert
certificate into the keystore:
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt
At this stage, all the switches should look familiar...
You should see the message:
Certificate was added to keystore
Then import the cert-signed
signed certificate into the keystore:
keytool -keystore kafka.server.keystore.jks -import -file cert-signed -storepass confluent -keypass confluent -noprompt
You should see:
Certificate reply was installed in keystore
vim kafka/config/server.properties
You'll be adding the following lines:
# TLS / SSL
ssl.keystore.location=/home/ubuntu/ssl/kafka.server.keystore.jks
ssl.keystore.password=confluent
ssl.key.password=confluent
ssl.truststore.location=/home/ubuntu/ssl/kafka.server.truststore.jks
ssl.truststore.password=confluent
You also need to ensure you have both listeners registered (the PLAINTEXT listener on port 9092 and the SSL listener on port 9093). You need to configure the listeners
property in order for this mapping to be created in the broker.
You also need to extend advertised.listeners
to add the new SSL listener; note that we're specifying the SSL port for your SSL connection in the advertised.listeners
property:
listeners=PLAINTEXT://:9092,SSL://:9093
advertised.listeners=PLAINTEXT://host-dns-name.compute.amazonaws.com:9092,SSL://host-dns-name.compute.amazonaws.com:9093
Now we need to restart Kafka:
sudo systemctl restart kafka && sudo systemctl status kafka
We want to see that the broker is in a running state after the configuration changes have been made:
● kafka.service - Apache Kafka server (broker)
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2022-04-08 12:52:32 UTC; 14ms ago
Verify connectivity by running:
openssl s_client -connect host-dns-name.compute.amazonaws.com:9093 -cipher ALL
You should now see something like:
CONNECTED(00000003)
depth=1 CN = Kafka-Security-CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=1 CN = Kafka-Security-CA
verify return:1
depth=0 CN = host-dns-name.compute.amazonaws.com
verify return:1
---
Grep for:
grep "endpoint" kafka/logs/server.log
You should see the SSL listener reported as "Started":
[2022-04-09 08:41:37,820] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane acceptor and processor(s) for endpoint : ListenerName(PLAINTEXT) (kafka.network.SocketServer)
[2022-04-09 08:41:37,849] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Started data-plane acceptor and processor(s) for endpoint : ListenerName(SSL) (kafka.network.SocketServer)
keytool -keystore ssl/kafka.client.keystore.jks -list -v
Note that Criticality=false in some cases. BasicConstraints CA:true (self signed)
certs-create-per-user.sh
All the commands back-to-back:
keytool -genkey -keystore kafka.server.keystore.jks -validity 365 -storepass confluent -keypass confluent -dname "CN=host-dns-name.compute.amazonaws.com" -storetype pkcs12 -keyalg RSA
keytool -keystore kafka.server.keystore.jks -certreq -file cert-file -storepass confluent -keypass confluent
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:confluent
keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt -keyalg RSA
keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt
keytool -keystore kafka.server.keystore.jks -import -file cert-signed -storepass confluent -keypass confluent -noprompt
sudo systemctl restart kafka && sudo systemctl status kafka
- openssl s_client -connect host-dns-name.compute.amazonaws.com:9093 -CAfile ssl/ca-cert -tls1_2
- openssl s_client -debug -connect host-dns-name.compute.amazonaws.com:9093
First ensure that you have the ca-cert
file (the public CA certificate) on your machine - in our case, we created this certificate on our remote Kafka broker machine, so we will need to scp
it over:
scp -i ~/key.pem [email protected]:~/ssl/ca-cert .
Check the directory on the machine that will be used for the client application for the ca-cert
file:
-rw-r--r-- 1 ableasdale staff 1.8K 10 Apr 09:01 ca-cert
- ca-cert is the public CA certificate
- alias is the CA Certificate alias name - we're using the same CARoot alias that was used previously
This will create your truststore file
keytool -keystore kafka.client.truststore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt -keyalg RSA
You should see the message:
Certificate was added to keystore
Check the keystore file:
keytool -list -v -keystore kafka.client.truststore.jks
You will be prompted for the password and then you should see something like this:
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
Note the Alias name and the Owner:
Alias name: caroot
[...]
Owner: CN=Kafka-Security-CA
Issuer: CN=Kafka-Security-CA
This will create your keystore file (kafka.client.keystore.jks
):
keytool -genkey -keystore kafka.client.keystore.jks -validity 365 -storepass confluent -keypass confluent -dname "CN=mylaptop" -alias my-local-pc -storetype pkcs12 -keyalg RSA
This will create the client-cert-sign-request
file for your client application:
keytool -keystore kafka.client.keystore.jks -certreq -file client-cert-sign-request -alias my-local-pc -storepass confluent -keypass confluent
In a production situation, this signing request would be sent to your Certificate Admin for signing. However, as before, we're going to rely on self-signing.
scp -i ~/key.pem client-cert-sign-request [email protected]:~/ssl
You should now see the client-cert-sign-request
in the ssl directory on the broker:
-rw-r--r-- 1 ubuntu ubuntu 977 Apr 10 07:19 client-cert-sign-request
ssh -i key.pem [email protected]
cd to the ssl
directory
Remember:
- ca-cert is the public CA certificate
- ca-key is the private key for the CA certificate
- in is the input file (
client-certificate-sign-request
) - out is the output file (
client-cert-signed
)
As with the broker section, we will use the openssl
commandline tool to sign the client certificate:
openssl x509 -req -CA ca-cert -CAkey ca-key -in client-cert-sign-request -out client-cert-signed -days 365 -CAcreateserial -passin pass:confluent
You should see:
Signature ok
subject=CN = mylaptop
Getting CA Private Key
Your directory listing will now contain the new client-cert-signed
certificate:
-rw-rw-r-- 1 ubuntu ubuntu 1346 Apr 10 07:29 client-cert-signed
We're going to use scp to copy the client certificate back to the machine running the client applications (in this case, it's a laptop):
From your client machine (i.e. your laptop), run:
scp -i ~/key.pem [email protected]:~/ssl/client-cert-signed .
You'll now see this in the directory:
-rw-r--r-- 1 ableasdale staff 1.3K 10 Apr 08:55 client-cert-signed
Now you can add the public CA certificate into the client keystore. We SCPd this file over earlier in the process.
Notes:
- keystore is the client keystore file (we're adding the root certificate to the store)
- alias this is the CA Certificate so we're using the same CARoot alias that was used previously
- file is the public CA certificate
keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file ca-cert -storepass confluent -keypass confluent -noprompt
You should see:
Certificate was added to keystore
Now we add the client signed certificate (client-cert-signed
) using the keytool
command:
keytool -keystore kafka.client.keystore.jks -import -file client-cert-signed -alias my-local-pc -storepass confluent -keypass confluent -noprompt
You should see:
Certificate reply was installed in keystore
Now we need to edit the server.properties
on the broker:
vim ~/kafka/config/server.properties
We need to add the following line to the broker's properties file:
# Client Authentication
ssl.client.auth=required
Restart Kafka to apply the new configuration changes:
sudo systemctl restart kafka && sudo systemctl status kafka
On your client machine, you need to create a properties file:
vim client-ssl-auth.properties
We're adding the following lines to the client-ssl-auth.properties
properties file:
security.protocol=SSL
ssl.truststore.location=~/ssl/kafka.client.truststore.jks
ssl.truststore.password=confluent
ssl.keystore.location=~/ssl/kafka.client.keystore.jks
ssl.keystore.password=confluent
ssl.key.password=confluent
We're using kafka-console-producer.sh
in the bin
directory of Apache Kafka and we're passing in the client-ssl-auth.properties
that we created in the previous step:
~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-producer.sh --broker-list host-dns-name.compute.amazonaws.com:9093 --topic kafka-security-topic --producer.config client-ssl-auth.properties
We're using kafka-console-consumer.sh
in the bin
directory of Apache Kafka and we're passing in the client-ssl-auth.properties
that we created in the previous step:
~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server host-dns-name.compute.amazonaws.com:9093 --topic kafka-security-topic --consumer.config client-ssl-auth.properties
Note that we have configured our broker with two listeners; there's the plaintext listener (on port 9092
) and the SSL listener (on port 9093
). We can still use the non-TLS endpoint to communicate with the brokers:
~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-producer.sh --broker-list host-dns-name.compute.amazonaws.com:9092 --topic kafka-security-topic
~/Documents/workspace/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh --bootstrap-server host-dns-name.compute.amazonaws.com:9092 --topic kafka-security-topic