Connected Cluster using k3s

k3s.adoc

Setup Guide

Table of Contents

• 1. OS packages
• 2. k3s
  • 1. Install k3s Server
  • 2. Install k3s Agent (Optional, only for multi-node cluster)
• 3. Connected Cluster

1. OS packages

First uninstall snapd (causes issues sometimes):

sudo apt purge -y snapd

There are some OS packages which can be useful:

• gparted - a graphical partition editor for creating, deleting, resizing, moving, checking, and copying disk partitions and the file systems on them.

• openssh-server - a secure shell (SSH) server which allows secure remote access to the system.

• git - needed for install linuxbrew (homebrew on linux).

• firefox - a web browser.

sudo apt update
sudo apt install -y gparted openssh-server git firefox curl build-essential

Install linuxbrew:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

echo 'eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"' >> ~/.bashrc
eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"

Install Azure CLI:

brew install azure-cli

2. k3s

Reference

• Each node needs to have a distinct hostname. If hostname is distinct, we can skip the K3S_NODE_NAME parameter below (in both server and agent installation commands).

• INSTALL_K3S_SKIP_SELINUX_RPM is needed to avoid SELinux installation.

• K3S_KUBECONFIG_MODE is needed to set the kubeconfig file permissions. By default, it is 600, but we need to set it to 644 so that it can be copied to other machines.

• INSTALL_K3S_EXEC is needed to disable traefik and enable secrets encryption.

1. Install k3s Server

This is the control plane node.

curl -sfL https://get.k3s.io | K3S_NODE_NAME="k3s-control-plane" K3S_KUBECONFIG_MODE="644" INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_EXEC='server --secrets-encryption --disable traefik' sh -

Meaning of each parameter:

curl -sfL https://get.k3s.io | \\
  K3S_NODE_NAME="k3s-control-plane" \\ // (1)
  K3S_KUBECONFIG_MODE="644" \\ // (2)
  INSTALL_K3S_SKIP_SELINUX_RPM=true \\ // (3)
  INSTALL_K3S_EXEC='server \\ // (4)
  --secrets-encryption \\ // (5)
  --disable traefik' \\ // (6)
  sh -

1. K3S_NODE_NAME is the name of the node. If not provided, the hostname will be used.

2. K3S_KUBECONFIG_MODE is the permission mode for the kubeconfig file. By default, it is 600, but we need to set it to 644 so that it can be copied to other machines.

3. INSTALL_K3S_SKIP_SELINUX_RPM is needed to avoid SELinux installation.

4. INSTALL_K3S_EXEC has various other options. server means this will be the control plane node.

5. --secrets-encryption is needed to enable secrets encryption.

6. --disable traefik is needed to disable traefik (the default ingress controller / application proxy gateway).

Verify the installation:

sudo k3s kubectl get nodes
sudo k3s kubectl get pods -A

Copy the default kubeconfig to the default location, and update the IP address to the node’s IP address:

mkdir -p ~/.kube
sudo cp /etc/rancher/k3s/k3s.yaml ~/.kube/config // (1)
sudo chown "$(id -u)":"$(id -g)" ~/.kube/config // (2)
NODE_IP="$(hostname -I | awk '{print $1}')" // (3)
sed -i "s/127.0.0.1/$NODE_IP/g" ~/.kube/config // (4)

1. Copy the k3s kubeconfig to the default kubeconfig location.

2. Change the ownership of the kubeconfig file to the current user.

3. Get the primary IP address of the node.

4. Update the kubeconfig file with the node’s IP address (instead of 127.0.0.1). This is needed so that the kubeconfig can be used from other machines.

After this, kubectl get pods -A should work too since ~/.kube/config is now pointing to the k3s server. sudo is not needed for this command since we changed the ownership of the kubeconfig file.

This kubeconfig can now be copied to any machine which has line-of-sight to the k3s server, and can be used with kubernetes clients

To create a multi-node cluster, we need to note down a few values from the server node:

1. The server node’s IP address

   NODE_IP="$(hostname -I | awk '{print $1}')"
   echo $NODE_IP

2. The server node’s token

   K3S_TOKEN=$(sudo cat /var/lib/rancher/k3s/server/node-token)

2. Install k3s Agent (Optional, only for multi-node cluster)

This is the worker node (name: worker-node-1).

NODE_NAME="worker-node-1" // (1)
NODE_IP="<NODE_IP>" // (2)
K3S_TOKEN="<K3S_TOKEN>" // (3)
curl -sfL https://get.k3s.io | K3S_NODE_NAME="$K3S_NODE_NAME" INSTALL_K3S_SKIP_SELINUX_RPM=true K3S_URL="https://$NODE_IP:6443" K3S_TOKEN="$K3S_TOKEN" sh -

1. NODE_NAME is the name of the node. If not provided, the hostname will be used.

2. NODE_IP is the IP address of the server node (noted in the previous step).

3. K3S_TOKEN is the token of the server node (noted in the previous step).

3. Connected Cluster

Install the Azure CLI extensions:

az extension add --name connectedk8s
az extension add --name k8s-extension
az extension add --name customlocation

Login to Azure:

az login --tenant "<tenant-id>" --use-device-code

Connect the cluster to Azure:

Note	--kube-config parameter is optional, if not provided, kubeconfig will be read from ~/.kube/config.

az connectedk8s connect --location <location> --resource-group <resource-group-name> --name <cluster-name> --kube-config ~/.kube/config