deploy_rancher_with_certs.md

setting up Rancher with certs - example

Docs : https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/resources/add-tls-secrets

install rke2

curl -sfL https://get.rke2.io |  sh -

set up env

echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=$PATH:/usr/local/bin/:/var/lib/rancher/rke2/bin/" >> ~/.bashrc
source ~/.bashrc

install helm

curl -s https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

add repos

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest --force-update 
helm repo add jetstack https://charts.jetstack.io --force-update 

install cert-manager

helm upgrade -i cert-manager jetstack/cert-manager -n cert-manager --create-namespace --set crds.enabled=true 

add secrets

kubectl create ns cattle-system

kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=/root/star.rfed.io.cert --key=/root/star.rfed.io.key

kubectl -n cattle-system create secret generic tls-ca --from-file=/root/cacerts.pem 

install rancher with tls certs

helm upgrade -i rancher rancher-latest/rancher -n cattle-system --create-namespace --set hostname=rancher.rfed.io --set bootstrapPassword=bootStrapAllTheThings --set replicas=1 --set ingress.tls.source=secret --set ingress.tls.secretName=tls-rancher-ingress --set privateCA=true 

the only difference between the two commands is here the cacerts.pem is located right?

Yeap, literally one is using /root/cacerts.pem and the other is using just cacerts.pem

It might be helm not parsing the CLI arguments correctly, but I have not dug in that far to validate.

huh... what version of rke2 and kubectl are you using?

have you updated helm as well?

Pulled from rancher manager
Provider: RKE2 Kubernetes Version: v1.30.5 +rke2r1 Architecture: Amd64 Created: 1.1 hours ago

kubectl
root@rancher-uk-01:~# kubectl version Client Version: v1.30.5+rke2r1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.5+rke2r1

I did not manually update helm after installing it using curl -s https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

Helm Version
root@rancher-uk-01:~# helm version version.BuildInfo{Version:"v3.16.2", GitCommit:"13654a52f7c70a143b1dd51416d633e1071faffb", GitTreeState:"clean", GoVersion:"go1.22.7"}

that looks all current. odd. can you recreate it easily?

Sure, let me spin up a new VM and I will follow the steps again.

So after creating 2 new VM's I can say that my initial issue was some odd fluke/random issue.

It works fine on both fresh VM with /root/cacerts.pem after copying the same commands from the first node
Weird, not sure what happened with the previous node considering I am using the same commands

However the file has to be named cacerts.pem if it is anything else it will error out with failed to setup TLS listener: read /etc/rancher/ssl/cacerts.pem: is a directory like before. (This was further testing)

Thanks for the help.

glad it worked out. I have seen similar oddities.

Hi Andy, how are you doing? Hope you're good, man :-)

Andy, the guys deployd a new root CA here at the company, so I'll have to update the certificate on my Rancher environment, and I taking a look on how to do it.
As this is the first time I'm doing that, I'm looking into the documentation (https://ranchermanager.docs.rancher.com/v2.8/getting-started/installation-and-upgrade/resources/update-rancher-certificate), and also found your video on YouTube as well (thanks for that, really helpful, Andy, not to mention the funny moment you tell you forgot the cert files haha)

One question that I have is when I get to that part related to update the CA certificate secret object (cacerts.pem)...
I've got the new root CA .pfx file, so in order to have the cacerts.pem I'll have to convert the .pfx file into the .pem format.
I believe that the command below (with the "-nodes" parameter) is the one I need to do that conversion, right?
openssl pkcs12 -in mycert.pfx -out cacerts.pem -nodes

If you could tell how you got your cacerts.pem file, I would really appreciate it.

Best regards,
toti

Hey toti! Things are good. Glad the guides are helpful. According to stackoverflow : https://stackoverflow.com/questions/15413646/converting-pfx-to-pem-using-openssl Yes it does look like that command should work!
I get my cacerts.pem from my Cert provider. It helps that I do almost everything in linux.

Oh great Andy, thank you very much!
Don't have so much time today, but will gonna try tomorrow.
+1 like/subscribed on your YT channel
See ya!

Awesome thank you! Let me know how it goes.

Hi @clemenko ,

Thank you very much for the guidance & video. Again, those are very helpful.

I have followed your tutorial with a bit customization in the SSL certificate (using letsEncrypt):

helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=rancher.awesome.com --set bootstrapPassword=pleas-change-me --set ingress.tls.source=letsEncrypt --set [email protected] --set letsEncrypt.ingress.class=nginx

But unfortunately got an issue reported here: rancher/rke2#7433 .

What do you think?

Brandon is really sharp. map out on that thread how your network is laid out.

Hi @andisugandi,

I know this is probably unrelated to your connection timeout issue with the Rancher endpoint, but I wanted to share a recommendation regarding the method you used to install the certificates.

I’ve tried a similar approach in the past, and while it wasn’t the cause of your specific issue, it did fail for me due to Rancher not receiving the full certificate chain. This led to problems with TLS handshake validation.

Instead, I’d highly recommend following the method outlined in [Techno Tim’s guide](https://technotim.live/posts/kube-traefik-cert-manager-le/) for setting up wildcard certificates with Traefik and cert-manager. His guide walks you through using DNS validation with your DNS provider’s API token to request certificates directly from Let’s Encrypt. You can even practice with staging certificates, but I’d suggest going “full send” and using production.

Why This Matters:

Rancher handles certificates differently and requires the full certificate chain for proper TLS validation. If this chain isn’t provided, it can lead to handshake failures. To ensure everything works smoothly, you need to prepare your certificates and secrets before installing Rancher.

The Correct Process:

Assuming you’ve followed Tim’s guide and cert-manager has issued certificates, you should have a certificate and corresponding secret in your namespace. For example:

➜ kubectl get certificates
NAME       READY   SECRET         AGE
yorko-io   True    yorko-io-tls   11m

➜ kubectl get secrets
NAME           TYPE                DATA   AGE
yorko-io-tls   kubernetes.io/tls   2      13m

Now, here’s how you prepare these certificates for Rancher:

Steps:

1. Create the namespace for Rancher:

   kubectl create namespace cattle-system

2. Add the Rancher Helm repository and update it:

   helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
   helm repo update

3. (Optional) Generate a Rancher admin password:

   RANCHER_PASSWORD=$(openssl rand -base64 12)

4. Extract the tls.crt from your cert-manager secret:

   kubectl get secret yorko-io-tls -n default -o jsonpath='{.data.tls\.crt}' | base64 --decode > tls.crt

5. Extract the tls.key from your cert-manager secret:

   kubectl get secret yorko-io-tls -n default -o jsonpath='{.data.tls\.key}' | base64 --decode > tls.key

6. Download the Let’s Encrypt CA certificate:

   curl -o letsencrypt-ca.pem https://letsencrypt.org/certs/isrgrootx1.pem

7. Combine the domain certificate and CA certificate:

   cat tls.crt > combined-tls.crt && cat letsencrypt-ca.pem >> combined-tls.crt

8. Create a Kubernetes secret for the CA certificate:

   kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem=letsencrypt-ca.pem --dry-run=client -o yaml | kubectl apply -f -

9. Create or update the Rancher TLS secret for the ingress:

   kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=combined-tls.crt --key=tls.key --dry-run=client -o yaml | kubectl apply -f -

10. Install or upgrade Rancher using Helm:

    helm upgrade -i rancher rancher-latest/rancher -n cattle-system --create-namespace \
      --set hostname=rancher.yorko.io \
      --set bootstrapPassword=bootStrapAllTheThings \
      --set replicas=1 \
      --set ingress.tls.source=secret \
      --set ingress.tls.secretName=tls-rancher-ingress \
      --set privateCA=true

Why This Way Works:

Rancher requires the full certificate chain to be presented during the TLS handshake. The single domain certificate issued by Let’s Encrypt doesn’t include the intermediate CA, which some clients need to validate the connection. By combining the certificates and properly configuring the Helm chart, you ensure compatibility across all clients and prevent TLS handshake issues.

I hope this helps resolve any certificate-related issues and simplifies your setup for the future! Let me know if you have questions.