HA K3S with kube-vip

how-to-k3s-ha.md

The Architecture

Below is a sample architecture, no workers for this example :-)

• VIP 192.168.0.40
• K8S01 192.168.0.41
• K8S02 192.168.0.42
• DB01 192.168.0.43

Create our DB on DB01

mkdir mysql
sudo docker run --cap-add SYS_NICE -p 3306:3306 --name k3s-mysql -v /home/$USER/mysql:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=k3s-password -d mysql:8 

Configure our first node and instantiate the VIP daemonset manifest

export VIP=192.168.0.40 
sudo mkdir -p /var/lib/rancher/k3s/server/manifests/
sudo docker run --network host --rm plndr/kube-vip:0.1.9 manifest daemonset \
--interface ens160 \
--vip $VIP \
--arp \
--leaderElection  |  sed 's/path: \/etc\/kubernetes\/admin.conf/path: \/etc\/rancher\/k3s\/k3s.yaml/g' |  sudo tee /var/lib/rancher/k3s/server/manifests/vip.yaml

Deploy K3s on both nodes

sudo ./k3s server --tls-san $VIP --datastore-endpoint="mysql://root:k3s-password@tcp(192.168.0.43:3306)/kubernetes"

Get a kubeconfig that uses the VIP

mkdir -p $HOME/.kube
sudo cat /etc/rancher/k3s/k3s.yaml | sed 's/127.0.0.1/'$VIP'/g' > $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Enjoy :-D

Extra

Below is the daemonset manifest if you don't want to use docker to automate the creation of the manifest:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  creationTimestamp: null
  name: kube-vip-ds
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: kube-vip-ds
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: kube-vip-ds
    spec:
      containers:
      - args:
        - start
        env:
        - name: vip_arp
          value: "true"
        - name: vip_interface
          value: ens160
        - name: vip_address
          value: 192.168.0.40
        - name: vip_leaderelection
          value: "true"
        - name: vip_leaseduration
          value: "5"
        - name: vip_renewdeadline
          value: "3"
        - name: vip_retryperiod
          value: "1"
        image: plndr/kube-vip:0.1.9
        imagePullPolicy: Always
        name: kube-vip
        resources: {}
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - SYS_TIME
        volumeMounts:
        - mountPath: /etc/kubernetes/admin.conf
          name: kubeconfig
        - mountPath: /etc/ssl/certs
          name: ca-certs
          readOnly: true
      hostNetwork: true
      nodeSelector:
        node-role.kubernetes.io/master: ""
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      volumes:
      - hostPath:
          path: /etc/rancher/k3s/k3s.yaml
        name: kubeconfig
      - hostPath:
          path: /etc/ssl/certs
        name: ca-certs
  updateStrategy: {}

In my case, using the daemonset manifest, I had to adjust the nodeSelector to:

nodeSelector:
  node-role.kubernetes.io/master: "true"

for the workload to be scheduled.

$ k3s --version
k3s version v1.19.2+k3s1 (d38505b1)