Rancher 2 HA using Helm and self signed certificate (certificate from files)

rancher2-ha-helm-selfsigned-certificate.md

Rancher 2 HA using Helm and self signed certificate (certificate from files)

This will only cover the part of installing Rancher on a RKE built cluster, see https://rancher.com/docs/rancher/v2.x/en/installation/ha/ how to get there.

Note: make sure kubeconfig is configured correctly

The commands are for Linux, if you are using Mac then you can use md5 instead of md5sum and base64 -D instead of base64 -d.

Generate certificates

In this example, we use a simple utility to create self signed certificates based on paulczar/omgwtfssl.

DOMAIN=rancher.mydomain.com
curl https://gist.githubusercontent.com/superseb/b2c1d6c9baa32609a49ee117a27bc700/raw/7cb196e974e13b213ac6ec3105971dd5e21e4c66/selfsignedcert.sh | bash -s -- $DOMAIN

This will place the certificates in /certs in the current working directory.

Install Rancher

Check if these instructions are still up2date with the instructions in the linked page above.

DOMAIN=rancher.mydomain.com
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
kubectl create namespace cattle-system
helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=$DOMAIN --set ingress.tls.source=secret --set privateCA=true
kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=certs/cert.pem --key=certs/key.pem
cp certs/ca.pem certs/cacerts.pem
kubectl -n cattle-system create secret generic tls-ca --from-file=certs/cacerts.pem

Wait for Rancher to be ready

kubectl -n cattle-system rollout status deploy/rancher
...
deployment "rancher" successfully rolled out

Check certificates

DOMAIN=rancher.mydomain.com
docker run superseb/ranchercheck https://$DOMAIN
...
Certificate chain is complete, connection to https://rancher.mydomain.com established successfully

Debugging

Check mounted file inside rancher/rancher containers

for pod in $(kubectl -n cattle-system get pods -l app=rancher -o custom-columns=NAME:.metadata.name --no-headers); do echo "Pod: $pod"; kubectl -n cattle-system exec $pod -- /bin/bash -c "cat /etc/rancher/ssl/cacerts.pem | openssl x509 -noout -subject -issuer -dates"; done

for pod in $(kubectl -n cattle-system get pods -l app=rancher -o custom-columns=NAME:.metadata.name --no-headers); do echo "Pod: $pod"; kubectl -n cattle-system exec $pod -- /bin/bash -c "cat /etc/rancher/ssl/cacerts.pem | md5sum"; done

Check created tls-ca secret

kubectl -n cattle-system get secret tls-ca -o "jsonpath={.data['cacerts\.pem']}" | base64 -d | openssl x509 -noout -subject -issuer -dates

kubectl -n cattle-system get secret tls-ca -o "jsonpath={.data['cacerts\.pem']}" | base64 -d | md5sum

Check /v3/settings/cacerts

Requires jq

curl -sLk https://rancher.mydomain.com/v3/settings/cacerts | jq -r .value | openssl x509 -noout -subject -dates -issuer

curl -sLk https://rancher.mydomain.com/v3/settings/cacerts | jq -r .value | md5sum

Check caCerts inside listenconfig cli-config

kubectl get listenconfig.management.cattle.io cli-config -o "jsonpath={.caCerts}" | openssl x509 -noout -subject -issuer -dates

kubectl get listenconfig.management.cattle.io cli-config -o "jsonpath={.caCerts}" | md5sum

When switching certificates, the value of caCerts can be stuck to the old value, and this will be copied to settings->cacerts. To sync them up, run:

SINGLELINECERT=$(kubectl -n cattle-system get secret tls-ca -o "jsonpath={.data['cacerts\.pem']}"  | base64 -d | awk 1 ORS='\\n')
kubectl patch listenconfig.management.cattle.io cli-config --type=merge --patch='{"caCerts": "'"$SINGLELINECERT"'"}'

I just have only one question. Why you generate two secrets? One is for NGINX ingress controller. And the second is for Rancher API and is needed for example when you add next Kubernetes cluster in Rancher using agents? Do I understand it correctly?

Yes, see https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/resources/add-tls-secrets

Thanks a lot!