Helpful links:

Preparations

We need all machines up and running. At first we will setup the loadbalancer.

Note:

Keep in mind to use rke2-killall.sh if you need to restart the service. You also may need to delete /var/lib/rancher/rke/server/db before setting up the cluster again.

Loadbalancer Nginx Setup

Configure /etc/hosts

$ cat /etc/hosts

####################################################################
#   IP                     FQDN                     ALIASES
#-------------- --------------------------- ------------------------

# Loadbalancer
<lb ip>                 <lb fqdn>                   <lb alias>

# K8s Server
<k8s server 1 ip>       <k8s server 1 fqdn>         CP_NODE_1 # <- We need the alias names in the nginx config
<k8s server 2 ip>       <k8s server 1 fqdn>         CP_NODE_2
<k8s server 3 ip>       <k8s server 1 fqdn>         CP_NODE_3

Configure /etc/nginx/nginx.conf

Now lets setup the nginx config:

user nginx;
worker_processes 4;
worker_rlimit_nofile 40000;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;


include /etc/nginx/modules-enabled/*.conf;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 8192;
}


http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;

##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

stream {
upstream backend {
        least_conn;
        server CP_NODE_1:9345 max_fails=3 fail_timeout=5s;
        server CP_NODE_2:9345 max_fails=3 fail_timeout=5s;
        server CP_NODE_3:9345 max_fails=3 fail_timeout=5s;
   }

   # This server accepts all traffic to port 9345 and passes it to the upstream.
   # Notice that the upstream name and the proxy_pass need to match.
server {
      listen 9345;
          proxy_pass backend;
   }
upstream ef_cx_api {
        least_conn;
        server CP_NODE_1:6443 max_fails=3 fail_timeout=5s;
        server CP_NODE_2:6443 max_fails=3 fail_timeout=5s;
        server CP_NODE_3:6443 max_fails=3 fail_timeout=5s;
    }
server {
        listen     6443;
        proxy_pass ef_cx_api; 
        }   
}

And (re)start the server systemctl restart nginx.

Prepare the k8s Cluster Servers

Now, before we start our cluster, all machines must be prepared. As a first step, lets create the necessary config files and directories.

mkdir -p /etc/rancher/rke2/
mkdir -p  /var/lib/rancher/rke2/server/manifests/

and now we create the rke config (remember to change ips and fqdn):

cat<<EOF|tee /etc/rancher/rke2/config.yaml
tls-san:
  # Loadbalancer
  - devops67.ef.com
  - 10.192.168.67
  # Cluster
  - devops61.ef.com
  - 10.192.168.61
  - devops62.ef.com
  - 10.192.168.62
  - devops63.ef.com
  - 10.192.168.63

EOF

Next create the nginx ingress config:

cat<<EOF| tee /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      metrics:
        service:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "10254"
      config:
        use-forwarded-headers: "true"
      allowSnippetAnnotations: "true"
EOF

Now we install rke2:

curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -

And start the service (ONE BY ONE!):

systemctl start rke2-server
systemctl enable rke2-server

Now add the kubeconfig and binaries path:

echo "export PATH=$PATH:/var/lib/rancher/rke2/bin" >> $HOME/.bashrc
echo "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml"  >> $HOME/.bashrc

And as the last step get the token for the other Cluster Nodes to join:

cat /var/lib/rancher/rke2/server/node-token

Configuring the other Cluster Nodes

We start again by creating the directories:

mkdir -p /etc/rancher/rke2/
mkdir -p  /var/lib/rancher/rke2/server/manifests/

and creaing the rke config file:

cat<<EOF|tee /etc/rancher/rke2/config.yaml
server: https://10.192.168.67:9345 # Loadbalancer ip
token: [token from /var/lib/rancher/rke2/server/node-token on server node 1]
tls-san:
  - devops67.ef.com
  - 10.192.168.67
  - devops61.ef.com
  - 10.192.168.61
  - devops62.ef.com
  - 10.192.168.62
  - devops63.ef.com
  - 10.192.168.63

EOF

And the nginx config again.

cat<<EOF| tee /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-config.yaml
---
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      metrics:
        service:
          annotations:
            prometheus.io/scrape: "true"
            prometheus.io/port: "10254"
      config:
        use-forwarded-headers: "true"
      allowSnippetAnnotations: "true"
EOF