Skip to content

Instantly share code, notes, and snippets.

@sebstyle

sebstyle/rancher-on-k3os.md

Last active March 21, 2024 03:43
Show Gist options
  • Save sebstyle/0a69f8bf18dd8c4aac73f3a25ff30739 to your computer and use it in GitHub Desktop.
Save sebstyle/0a69f8bf18dd8c4aac73f3a25ff30739 to your computer and use it in GitHub Desktop.
Download ZIP
Install rancher on k3os
Raw
rancher-on-k3os.md

Rancher cluster on k3os

Goal

Install rancher on k3os with centralized datastore on mysql cluster.

Environment

  • Host: Ubuntu 18.04.4 Desktop (32gb ram 4 cpu)
  • Virtualisation: kvm/qemu (virtual machine manager)
  • Local network: 192.168.0.0/16
    • Gateway: 192.168.0.1
    • Nameserver: 192.168.254.254
  • Local domain: vc.lan
    • dns.vc.lan: 192.168.254.254
    • db1.vc.lan: 192.168.5.1
    • db2.vc.lan: 192.168.5.2
    • rancher.vc.lan: 192.168.5.5
    • rancher1.vc.lan: 192.168.8.1
    • rancher2.vc.lan: 192.168.8.2

In my environment I have created a pdns-recursor on 1 virtual machine to service the vc.lan domain.

  • dns.vc.lan (1gb ram 1vcpu 4gb disk)

Create database server(s)

https://rancher.com/docs/rancher/v2.x/en/installation/k8s-install/create-nodes-lb/

In my environment I have created a mysql cluster (ubuntu 18 with mysql) on 2 virtual machines.

  • db1.vc.lan (1gb ram 2vcpu 4gb disk) (master)
  • db2.vc.lan (1gb ram 2vcpu 4gb disk) (slave)

Database uri: mysql://rancher:rancherpass@tcp(db1.vc.lan:3306)/rancher

Create loadbalancer

https://rancher.com/docs/rancher/v2.x/en/installation/options/nginx/

In my environment I have created a loadbalancer (ubuntu 18 with nginx) on 1 virtual machine.

  • rancher.vc.lan (1gb ram 2vcpu 4gb disk)

/etc/nginx/nginx.conf

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
}

http {
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        include /etc/nginx/mime.types;
        default_type application/octet-stream;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;
        gzip on;
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

stream {
    upstream rancher_servers_http {
        least_conn;
        server 192.168.20.1:80 max_fails=3 fail_timeout=5s;
        server 192.168.20.2:80 max_fails=3 fail_timeout=5s;
    }

    server {
        listen 80;
        proxy_pass rancher_servers_http;
    }

    upstream rancher_servers_https {
        least_conn;
        server 192.168.20.1:443 max_fails=3 fail_timeout=5s;
        server 192.168.20.2:443 max_fails=3 fail_timeout=5s;
    }

    server {
        listen     443;
        proxy_pass rancher_servers_https;
    }
}

Basicly the default nginx.conf with the stream definition added.

Modify default website on loadbalancer

/etc/nginx/sites-enabled/default

server {
        listen 8080 default_server;
        listen [::]:8080 default_server;
        root /var/www/html;
...
...
}

Create cloud-config files on the default website of the loadbalancer

/var/www/html/rancher1.yml

#cloud-config
hostname: rancher1
ssh_authorized_keys:
  - ssh-rsa [AUTHORIZED_KEY]

write_files:
- enconding: ""
  content: |-
    #!/bin/bash
    write_log () {
        local message="${1}"
        logger -t "run-cmd" "${message}"
        echo "${message}"
    }
    write_log "Getting the service using eth0..."
    ETH0=$(connmanctl services | awk '{ print $3 }' | while read -r s1; do connmanctl services $s1 | grep -q "eth0" && echo "$s1"; done)
    write_log "Device eth0 is configured using: ${ETH0}"
    write_log "Setting up manual net config..."
    connmanctl config $ETH0 --ipv4 manual 192.168.8.1 255.255.0.0 192.168.0.1 --nameservers 192.168.254.254 --domains vc.lan
    write_log "Network setup done."
  owner: root:root
  path: /etc/run-cmd.sh
  permissions: '0755'
run_cmd:
- "/etc/run-cmd.sh"

k3os:
  dns_nameservers:
    - 192.168.254.254
  ntp_servers:
    - 0.nl.pool.ntp.org
    - 1.nl.pool.ntp.org
  k3s_args:
    - server
    - "--datastore-endpoint=mysql://rancher:rancherpass@tcp(db1.vc.lan:3306)/rancher"

/var/www/html/rancher2.yml

#cloud-config
hostname: rancher2
ssh_authorized_keys:
  - ssh-rsa [AUTHORIZED_KEY]

write_files:
- enconding: ""
  content: |-
    #!/bin/bash
    write_log () {
        local message="${1}"
        logger -t "run-cmd" "${message}"
        echo "${message}"
    }
    write_log "Getting the service using eth0..."
    ETH0=$(connmanctl services | awk '{ print $3 }' | while read -r s1; do connmanctl services $s1 | grep -q "eth0" && echo "$s1"; done)
    write_log "Device eth0 is configured using: ${ETH0}"
    write_log "Setting up manual net config..."
    connmanctl config $ETH0 --ipv4 manual 192.168.8.2 255.255.0.0 192.168.0.1 --nameservers 192.168.254.254 --domains vc.lan
    write_log "Network setup done."
  owner: root:root
  path: /etc/run-cmd.sh
  permissions: '0755'
run_cmd:
- "/etc/run-cmd.sh"

k3os:
  dns_nameservers:
    - 192.168.254.254
  ntp_servers:
    - 0.nl.pool.ntp.org
    - 1.nl.pool.ntp.org
  k3s_args:
    - server
    - "--datastore-endpoint=mysql://rancher:rancherpass@tcp(db1.vc.lan:3306)/rancher"

Install k3os

Download the latest k3os:

Spin up a virtual machine:

  • rancher1.vc.lan (4gb ram 2vcpu 8gb disk)

Boot the virtual machine from the k3os-amd64 iso:

  • login with the rancher user
  • install to disk sudo k3os install
  • use the cloud-config files when the installer asks for them:
    • http://rancher.vc.lan:8080/rancher1.yml
  • reboot from disk

Do the same thing for the second node. (rancher2)

Download helm

Download the helm binary to your local machine:

  • wget https://get.helm.sh/helm-v3.2.0-linux-amd64.tar.gz
  • tar -zxvf helm-v3.2.0-linux-amd64.tar.gz
  • sudo mv linux-amd64/helm /usr/local/bin

Download kubectl

Download the kubectl binary to your local machine:

  • curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.18.0/bin/linux/amd64/kubectl
  • chmod +x kubectl
  • sudo mv kubectl /usr/local/bin

Note: https://kubernetes.io/docs/tasks/tools/install-kubectl/ for more info.

Download kubeconfig file

Download the kubeconfig file to your local machine:

  • scp [email protected]:/etc/rancher/k3s/k3s.yaml ~/rancher.kubeconfig
  • chmod 0600 ~/rancher.kubeconfig

You can now admin the cluster from your local machine.

Install cert-manager

Install cert-manager on the cluster using kubectl and helm:

  • helm repo add jetstack https://charts.jetstack.io
  • helm repo update
  • kubectl --kubeconfig ~/rancher.kubeconfig create namespace cert-manager
  • kubectl --kubeconfig ~/rancher.kubeconfig apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.14.3/cert-manager.crds.yaml
  • helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.14.3 --kubeconfig ~/rancher.kubeconfig

The cert-manager webhook pod must be operational before rancher can be installed:

  • kubectl --kubeconfig ~/rancher.kubeconfig rollout status -n cert-manager deploy/cert-manager-webhook

Note: https://cert-manager.io/docs/installation/kubernetes/ for more info.

Install rancher

Install rancher on the cluster using kubectl and helm:

  • helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
  • helm repo update
  • kubectl --kubeconfig ~/rancher.kubeconfig create namespace cattle-system
  • helm install rancher rancher-stable/rancher --namespace cattle-system --set hostname=rancher.vc.lan --kubeconfig ~/rancher.kubeconfig

Note: hostname=rancher.vc.lan is the hostname of the loadbalancer.

More nodes

To add more nodes (rancher3) repeat Install k3os for each additional node you want to add to the rancher cluster.

Bonus: custom cluster in rancher

  • Local domain: vc.lan
    • k8s1.vc.lan: 192.168.10.1
    • k8s2.vc.lan: 192.168.10.2

Create cloud-config files on the default website of the loadbalancer

/var/www/html/k8s1.yml

#cloud-config
hostname: k8s1
ssh_authorized_keys:
  - ssh-rsa [AUTHORIZED_KEY]

rancher:
  network:
    interfaces:
      eth0:
        address: 192.168.10.1/16
        gateway: 192.168.0.1
    dns:
      nameservers:
        - 192.168.254.254

/var/www/html/k8s2.yml

#cloud-config
hostname: k8s2
ssh_authorized_keys:
  - ssh-rsa [AUTHORIZED_KEY]

rancher:
  network:
    interfaces:
      eth0:
        address: 192.168.10.2/16
        gateway: 192.168.0.1
    dns:
      nameservers:
        - 192.168.254.254

Create a custom cluster in the rancher web interface

So easy!

Install rancheros

Download the latest rancheros:

Spin up a virtual machine:

  • k8s1.vc.lan (8gb ram 4vcpu 16gb disk)

Boot the virtual machine from the rancheros iso:

  • install to disk sudo ros install -d /dev/vda -c http://rancher.vc.lan:8080/k8s1.yml
  • reboot from disk

Deploy custom cluster managed by rancher

  • ssh [email protected]
  • docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.3.6 --server https://rancher.vc.lan --token [TOKEN] --ca-checksum [CHECKSUM] --etcd --controlplane --worker

Note: The docker run command is copy/pasted from the custom cluster setup page in the rancher web interface.

Do the same thing for the second node. (k8s2)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment