Externalizing the Trust Store for Hazelcast Enterprise Docker

A workaround for the current lack of HashiCorp or other vault support.

Objective:

To externalize the Java trust store from the Hazelcast Docker container, allowing updates to the trust store without the need to rebuild the Docker image.

Steps to Implement:

Create an External Volume for the Trust Store:
When running the Hazelcast Docker container, you can mount an external directory from the host system as a volume. This directory will hold the trust store (e.g., cacerts).
This setup allows you to update the trust store directly on the host system, and those changes will be reflected within the running container.

Configure Hazelcast to Use the External Trust Store:

Configure the Java runtime in the Docker container to use the externalized trust store by setting the javax.net.ssl.trustStore system property.
You can do this by adding the following Java options in the Hazelcast start command or Docker environment variables.

  -Djavax.net.ssl.trustStore=/opt/hazelcast/truststore/cacerts \
-Djavax.net.ssl.trustStorePassword=changeit

Update the Trust Store:

To update certificates, modify the trust store file in the external directory on the host system. For instance, you can add a new certificate using the keytool command:

keytool -import -trustcacerts -keystore /path/to/external/truststore/cacerts \
-storepass changeit -noprompt -alias new-cert -file /path/to/new/certificate.crt

Reload Certificates Without Restarting the Java Process:

Unfortunately, Java applications typically read the trust store once at startup, meaning updates to the trust store file won’t be picked up automatically without restarting the Java process.
As a result, while this method avoids rebuilding the Docker image, it may still require a restart of the Hazelcast process or container to load the new certificates.

Pros and Cons:

Pros:

No need to rebuild the Docker image every time a certificate is added or updated.
Simplifies the certificate update process by allowing modifications directly on the host system.

Cons:

The Hazelcast container or process might still need to be restarted to apply the updated certificates.
This solution is not fully dynamic as it doesn’t eliminate restarts, but it does reduce the operational complexity.

Sample Script

This will create a directory called ./external_truststore in your current working directory. As this is only useful for Hazelcast Enterprise, you will need a license key. This should be set with 'export HZ_LICENSEKEY=YOURKEY12345...'" Docker should be running. (Tested with Docker Desktop for Mac.)

#!/bin/bash

echo "This script relies on having your license key in an env var called HZ_LICENSEKEY:"

# Set the base project directory to the current working directory
CURRENT_DIR=$(pwd)
PROJECT_DIR="$CURRENT_DIR/external_truststore"
CERT_DIR="$PROJECT_DIR/certs"
TRUSTSTORE_DIR="$PROJECT_DIR/truststore"
TRUSTSTORE_PASS="changeit"
KEY_ALIAS="hz-cert"
CERT_FILE="$CERT_DIR/hz-cert.crt"
KEY_FILE="$CERT_DIR/hz-cert.key"
TRUSTSTORE_FILE="$TRUSTSTORE_DIR/cacerts"
LICENSE_KEY="$HZ_LICENSEKEY"

if [ -z "$HZ_LICENSEKEY" ]; then
  echo "License key not set. Please store your license string with 'export HZ_LICENSEKEY=YOURKEY12345...'"
  exit 1
fi

# Create cert and trust store dirs 
mkdir -p "$CERT_DIR"
mkdir -p "$TRUSTSTORE_DIR"

# Create private key and self-signed certificate
openssl req -x509 -newkey rsa:4096 -sha256 -days 365 -nodes \
    -keyout "$KEY_FILE" -out "$CERT_FILE" \
    -subj "/CN=hazelcast.local"

# Create trust store and import the certificate
keytool -importcert -trustcacerts -noprompt -alias "$KEY_ALIAS" \
    -file "$CERT_FILE" -keystore "$TRUSTSTORE_FILE" -storepass "$TRUSTSTORE_PASS"

# Run the first Hazelcast container with the external trust store mounted and the license key
docker run -d --name hazelcast-enterprise \
  -v "$TRUSTSTORE_DIR:/opt/hazelcast/truststore" \
  -e JAVA_OPTS="-Djavax.net.ssl.trustStore=/opt/hazelcast/truststore/cacerts -Djavax.net.ssl.trustStorePassword=$TRUSTSTORE_PASS" \
  -e HZ_LICENSEKEY="$LICENSE_KEY" \
  hazelcast/hazelcast-enterprise:latest

# Run another Hazelcast container to show keys and certs work
docker run -d --name second_hazelcast-enterprise \
  -v "$TRUSTSTORE_DIR:/opt/hazelcast/truststore" \
  -e JAVA_OPTS="-Djavax.net.ssl.trustStore=/opt/hazelcast/truststore/cacerts -Djavax.net.ssl.trustStorePassword=$TRUSTSTORE_PASS" \
  -e HZ_LICENSEKEY="$LICENSE_KEY" \
  hazelcast/hazelcast-enterprise:latest

echo "2 Hazelcast containers are running with the external trust store mounted at $TRUSTSTORE_DIR."

Add a key to the truststore and restart one of the containers:

This generates a new self-signed cert and adds it to the truststore, then it restarts the first container without needing to rebuild. If you wanted to use a completely new truststore, you would probably need to restart the whole cluster, but there would still be no need to rebuild the containers.

#!/bin/bash

# Set variables
PROJECT_DIR="/Users/jim/hz-support/00014761/external_truststore"
CERT_DIR="$PROJECT_DIR/certs"
TRUSTSTORE_DIR="$PROJECT_DIR/truststore"
TRUSTSTORE_PASS="changeit"
CERT_ALIAS="new-cert-$(date +%s)"
CERT_FILE="$CERT_DIR/$CERT_ALIAS.crt"
KEY_FILE="$CERT_DIR/$CERT_ALIAS.key"
TRUSTSTORE_FILE="$TRUSTSTORE_DIR/cacerts"
CONTAINER_NAME="hazelcast-enterprise"

# Create necessary directories if they don't exist
mkdir -p "$CERT_DIR"

# Generate a new private key and self-signed certificate
openssl req -x509 -newkey rsa:4096 -sha256 -days 365 -nodes \
    -keyout "$KEY_FILE" -out "$CERT_FILE" \
    -subj "/CN=hazelcast.local"

if [ $? -ne 0 ]; then
  echo "Failed to generate certificate."
  exit 1
fi

echo "Generated new certificate: $CERT_FILE"

# Import the new certificate into the trust store
keytool -importcert -trustcacerts -noprompt -alias "$CERT_ALIAS" \
    -file "$CERT_FILE" -keystore "$TRUSTSTORE_FILE" -storepass "$TRUSTSTORE_PASS"

if [ $? -ne 0 ]; then
  echo "Failed to import certificate."
  exit 1
fi

echo "Certificate '$CERT_ALIAS' successfully imported into trust store."

# Restart the Hazelcast container to apply the new certificate
docker restart "$CONTAINER_NAME"

if [ $? -eq 0 ]; then
  echo "Hazelcast process restarted successfully."
else
  echo "Failed to restart the Hazelcast process."
  exit 1
fi

# List all certificates in the trust store
echo "Listing all certificates in the trust store:"
keytool -list -keystore "$TRUSTSTORE_FILE" -storepass "$TRUSTSTORE_PASS"
'''

jimoconnell/README.md