, , _______________________________
,-----------|'------'| | |
/. '-' |-' |_____________________________|
|/| | |
| .________.'----' _______________________________
| || | || | |
\__|' \__|' |_____________________________|
|‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾|
|________________________________________________________|
|‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾|
|________________________________________________________|On-premise Kubernetes installations are unable to take advantage of cloud-native services like dynamic load-balancers. In order to ensure highly-available clusters, one must deploy a solution that will enable the Kubernetes API-server to be accessible in the event of node failure. While traditionally this would be accomplished using an on-premise load-balancer such as k8s-deployed metal-lb/nginx, these solutions would not work our case because the api-scheduler would not be available to schedule such deployments... therefore, chicken and the egg.
The kube-vip project provides High-Availability and load-balancing for both inside and outside a Kubernetes cluster
In order to proceed with this guide, you will need the following:
- DNS server or modification of
/etc/hostswith the node hostnames and rke2 master HA hostname firewalldturned off
In this guide, I will be setting up a 3-node HA RKE2 cluster. I use the .lol domain but swap out for the domain of your choosing.
| Host | Type | IP | Notes |
|---|---|---|---|
| rke2a | VM | 10.0.1.2 | etcd |
| rke2b | VM | 10.0.1.3 | etcd |
| rke2c | VM | 10.0.1.4 | etcd |
| rke2master | Virtual-IP | 10.0.1.5 | You will define this IP on your own. Make sure that it is not currently allocated to a node (and remove from DHCP allocation) |
If you do not have a DNS server available/configured, the /etc/hosts file on each node will need to include the following.
rke2a 10.0.1.2
rke2b 10.0.1.3
rke2c 10.0.1.4
rke2master 10.0.1.5The secret behind on-prem HA is kube-vip. We are going to modify their recommended approach with k3s and use it for rke2.
export RKE2_VIP_IP=10.0.1.5 # IMPORTANT: Update this with the IP that you chose.
# create RKE2's self-installing manifest dir
mkdir -p /var/lib/rancher/rke2/server/manifests/
# Install the kube-vip deployment into rke2's self-installing manifest folder
curl -sL kube-vip.io/k3s | vipAddress=${RKE2_VIP_IP} vipInterface=eth0 sh | sudo tee /var/lib/rancher/rke2/server/manifests/vip.yaml
# Find/Replace all k3s entries to represent rke2
sed -i 's/k3s/rke2/g' /var/lib/rancher/rke2/server/manifests/vip.yaml
# create the rke2 config file
mkdir -p /etc/rancher/rke2
touch /etc/rancher/rke2/config.yaml
echo "tls-san:" >> /etc/rancher/rke2/config.yaml
echo " - ${HOSTNAME}.lol" >> /etc/rancher/rke2/config.yaml
echo " - ${HOSTNAME}" >> /etc/rancher/rke2/config.yaml
echo " - rke2master.lol" >> /etc/rancher/rke2/config.yaml
echo " - rke2master" >> /etc/rancher/rke2/config.yaml
## Optional but recommended
# k9s - ncurses-based k8s dashboard
wget https://github.com/derailed/k9s/releases/download/v0.24.2/k9s_Linux_x86_64.tar.gz -O /tmp/k9s.tgz ; cd /tmp; tar zxvf k9s.tgz ; chmod +x ./k9s; mv ./k9s /usr/local/bin
# update path with rke2-binaries
echo 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml' >> ~/.bashrc ; echo 'export PATH=${PATH}:/var/lib/rancher/rke2/bin' >> ~/.bashrc ; echo 'alias k=kubectl' >> ~/.bashrc ; source ~/.bashrc ;curl -sfL https://get.rke2.io | sh -
systemctl enable rke2-server.service
systemctl start rke2-server.service
sleep 90 #wait ~90 seconds for rke2 to be ready
kubectl get nodes -o wide # should show as readyIn order for us to ensure that the virtual-ip is serving up the api-server, run the following commands:
mkdir -p $HOME/.kube
export VIP=rke2master
sudo cat /etc/rancher/rke2/rke2.yaml | sed 's/127.0.0.1/'$VIP'/g' > $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
KUBECONFIG=~/.kube/config kubectl get nodes -o wideThis will actually use the virtual-ip that kube-vip created for us at 10.0.1.5 and ensure that the rke2 api is successfully being served on that virtual-ip.
Well in order for our cluster to be highly available, naturally we are going to need to have additional masters.
On each of the additional masters, run the following commands:
# IMPORTANT: replace the following with the value of /var/lib/rancher/rke2/server/token from rke2a
export TOKEN=""
mkdir -p /etc/rancher/rke2
touch /etc/rancher/rke2/config.yaml
echo "token: ${TOKEN}" >> /etc/rancher/rke2/config.yaml
echo "server: https://rke2master.lol:9345" >> /etc/rancher/rke2/config.yaml
echo "tls-san:" >> /etc/rancher/rke2/config.yaml
echo " - ${HOSTNAME}.lol" >> /etc/rancher/rke2/config.yaml
echo " - ${HOSTNAME}" >> /etc/rancher/rke2/config.yaml
echo " - rke2master.lol" >> /etc/rancher/rke2/config.yaml
echo " - rke2master" >> /etc/rancher/rke2/config.yaml
curl -sfL https://get.rke2.io | sh -
systemctl enable rke2-server.service
systemctl start rke2-server.serviceThat's it! As you can see in the config file above, we are actually referencing the virtual-ip/host as the rke2 server and not a host's specific ip/host as we want the reference to move with the availability of etcd.
I have 3 master nodes. As long as 1/3 master nodes are alive, your cluster will keep functioning with no downtime.
If rke2a is the master and if it goes down, then another master will be chosen from the other 2 remaining node and the floating ip is assigned to the newly elected master. When the rke2a comes back, the election process reoccurs to choose a new master. As long as there is 1 master node alive, kube-vip will keep the k8s cluster alive.