, , _______________________________
,-----------|'------'| | |
/. '-' |-' |_____________________________|
|/| | |
| .________.'----' _______________________________
| || | || | |
\__|' \__|' |_____________________________|
|‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾|
|________________________________________________________|
|‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾|
|________________________________________________________|On-premise Kubernetes installations are unable to take advantage of cloud-native services like dynamic load-balancers. In order to ensure highly-available clusters, one must deploy a solution that will enable the Kubernetes API-server to be accessible in the event of node failure. While traditionally this would be accomplished using an on-premise load-balancer such as k8s-deployed metal-lb/nginx, these solutions would not work our case because the api-scheduler would not be available to schedule such deployments... therefore, chicken and the egg.
The kube-vip project provides High-Availability and load-balancing for both inside and outside a Kubernetes cluster
In order to proceed with this guide, you will need the following:
- DNS server or modification of
/etc/hostswith the node hostnames and rke2 master HA hostname firewalldturned off
In this guide, I will be setting up a 3-node HA RKE2 cluster. I use the .lol domain but swap out for the domain of your choosing.
| Host | Type | IP | Notes |
|---|---|---|---|
| rke2a | VM | 10.0.1.2 | etcd |
| rke2b | VM | 10.0.1.3 | etcd |
| rke2c | VM | 10.0.1.4 | etcd |
| rke2master | Virtual-IP | 10.0.1.5 | You will define this IP on your own. Make sure that it is not currently allocated to a node (and remove from DHCP allocation) |
If you do not have a DNS server available/configured, the /etc/hosts file on each node will need to include the following.
rke2a 10.0.1.2
rke2b 10.0.1.3
rke2c 10.0.1.4
rke2master 10.0.1.5The secret behind on-prem HA is kube-vip. We are going to modify their recommended approach with k3s and use it for rke2.
export RKE2_VIP_IP=10.0.1.5 # IMPORTANT: Update this with the IP that you chose.
# create RKE2's self-installing manifest dir
mkdir -p /var/lib/rancher/rke2/server/manifests/
# Install the kube-vip deployment into rke2's self-installing manifest folder
curl -sL kube-vip.io/k3s | vipAddress=${RKE2_VIP_IP} vipInterface=eth0 sh | sudo tee /var/lib/rancher/rke2/server/manifests/vip.yaml
# Find/Replace all k3s entries to represent rke2
sed -i 's/k3s/rke2/g' /var/lib/rancher/rke2/server/manifests/vip.yaml
# create the rke2 config file
mkdir -p /etc/rancher/rke2
touch /etc/rancher/rke2/config.yaml
echo "tls-san:" >> /etc/rancher/rke2/config.yaml
echo " - ${HOSTNAME}.lol" >> /etc/rancher/rke2/config.yaml
echo " - ${HOSTNAME}" >> /etc/rancher/rke2/config.yaml
echo " - rke2master.lol" >> /etc/rancher/rke2/config.yaml
echo " - rke2master" >> /etc/rancher/rke2/config.yaml
## Optional but recommended
# k9s - ncurses-based k8s dashboard
wget https://github.com/derailed/k9s/releases/download/v0.24.2/k9s_Linux_x86_64.tar.gz -O /tmp/k9s.tgz ; cd /tmp; tar zxvf k9s.tgz ; chmod +x ./k9s; mv ./k9s /usr/local/bin
# update path with rke2-binaries
echo 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml' >> ~/.bashrc ; echo 'export PATH=${PATH}:/var/lib/rancher/rke2/bin' >> ~/.bashrc ; echo 'alias k=kubectl' >> ~/.bashrc ; source ~/.bashrc ;curl -sfL https://get.rke2.io | sh -
systemctl enable rke2-server.service
systemctl start rke2-server.service
sleep 90 #wait ~90 seconds for rke2 to be ready
kubectl get nodes -o wide # should show as readyIn order for us to ensure that the virtual-ip is serving up the api-server, run the following commands:
mkdir -p $HOME/.kube
export VIP=rke2master
sudo cat /etc/rancher/rke2/rke2.yaml | sed 's/127.0.0.1/'$VIP'/g' > $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
KUBECONFIG=~/.kube/config kubectl get nodes -o wideThis will actually use the virtual-ip that kube-vip created for us at 10.0.1.5 and ensure that the rke2 api is successfully being served on that virtual-ip.
Well in order for our cluster to be highly available, naturally we are going to need to have additional masters.
On each of the additional masters, run the following commands:
# IMPORTANT: replace the following with the value of /var/lib/rancher/rke2/server/token from rke2a
export TOKEN=""
mkdir -p /etc/rancher/rke2
touch /etc/rancher/rke2/config.yaml
echo "token: ${TOKEN}" >> /etc/rancher/rke2/config.yaml
echo "server: https://rke2master.lol:9345" >> /etc/rancher/rke2/config.yaml
echo "tls-san:" >> /etc/rancher/rke2/config.yaml
echo " - ${HOSTNAME}.lol" >> /etc/rancher/rke2/config.yaml
echo " - ${HOSTNAME}" >> /etc/rancher/rke2/config.yaml
echo " - rke2master.lol" >> /etc/rancher/rke2/config.yaml
echo " - rke2master" >> /etc/rancher/rke2/config.yaml
curl -sfL https://get.rke2.io | sh -
systemctl enable rke2-server.service
systemctl start rke2-server.serviceThat's it! As you can see in the config file above, we are actually referencing the virtual-ip/host as the rke2 server and not a host's specific ip/host as we want the reference to move with the availability of etcd.
I really like this idea and want to try it out. However I have an existing cluster already.
Does this mean I cannot use it in an existing cluster?
Many thanks.