This is a 5-node Raspberry Pi 5 cluster, assembled from PicoCluster, with an integrated power source and 8-port network switch and cooling fan.
This means it can be plugged in to power with a single power cable, and you can connect to any of the 5 boards using a single ethernet cable.
The goal of this tutorial is to configure the Pi cluster to run Kubernetes. To do so, we need to:
- install an operating system on all the nodes
- enable cgroups to run containers on each node
- configure Kubernetes
Note: this process could be significantly improved by doing netbooting.
First, we need to download an appropriate operating system image from https://www.raspberrypi.com/software/operating-systems/.
If you need to run Wasm applications using SpinKube (or directly through Spin or Wasmtime), you need to install a 64-bit version of the operating system. I downloaded Raspberry Pi OS Lite 64-bit.
Then, you need to install the Raspberry Pi Imager software, which you can use to flash all SD cards for each Raspberry Pi.
A prerequisite for configuring Kubernetes on the cluster is setting up SSH key authentication for each board (you can follow the instructions here), and optionally, you can also setup headless WiFi access and credentials.
Note: when setting up each board, I setup the names pi0…pi4, which will become the names for each node in the cluster.
Once you flashed the SD cards, you need to insert the SD cards in the Pi boards, then power on the cluster.
Depending on whether you setup WiFi access or whether you plug the cluster (or each board individually) to ethernet, you can search your local network for the IP address of each board.
Assuming the boot process was successful, you can use arp-scan
(or your favorite network scanning tool) to search for the local network address. For example, this is the output for my local setup:
$ sudo arp-scan --interface=en1 --localnet
Password:
Interface: en1, type: EN10MB
Starting arp-scan 1.10.0 with 512 hosts (https://github.com/royhills/arp-scan)
192.168.129.49 d8:3a:dd:d2:24:67 Raspberry Pi Trading Ltd
192.168.129.50 d8:3a:dd:d2:22:8e Raspberry Pi Trading Ltd
192.168.129.52 d8:3a:dd:d2:25:1e Raspberry Pi Trading Ltd
192.168.129.48 d8:3a:dd:d2:23:d4 Raspberry Pi Trading Ltd
192.168.129.51 d8:3a:dd:d2:24:76 Raspberry Pi Trading Ltd
Once you have found the IP addresses of your boards, you can now SSH and find the ethernet IP addresses, which we can use to setup the cluster with.
$ ssh -i ~/path/to/ssh/key [email protected]
pi@pi0:~ $ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.2 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fd49:8b6e:788f:6d04:da3a:ddff:fed2:2466 prefixlen 64 scopeid 0x0<global>
inet6 fe80::da3a:ddff:fed2:2466 prefixlen 64 scopeid 0x20<link>
ether d8:3a:dd:d2:24:66 txqueuelen 1000 (Ethernet)
RX packets 48135 bytes 6775993 (6.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40750 bytes 25060917 (23.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 107
Note: while SSH-ing into each board, enable cgroups:
$ cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=f62dd3c1-02 rootfstype=ext4 fsck.repair=yes rootwait cfg80211.ieee80211_regdom=BE cgroup_memory=1 cgroup_enable=memory
Create a devices.json
file that contains the hostname and ethernet IP address for each board. For example, this is what my file looks like:
[
{
"hostname": "pi0",
"ip": "192.168.2.2"
},
{
"hostname": "pi1",
"ip": "192.168.2.5"
},
{
"hostname": "pi2",
"ip": "192.168.2.6"
},
{
"hostname": "pi3",
"ip": "192.168.2.3"
},
{
"hostname": "pi4",
"ip": "192.168.2.4"
}
]
Note: you will have to set static IP addresses for each board; not doing so might result in the cluster configuration being broken.
We can use [k3sup](https://github.com/alexellis/k3sup)
a tool written by Alex Ellis to easily configure k3s on my 5 node cluster. Using the devices.json
file, we use the k3sup plan
command to generate a script that will configure k3s on all nodes:
$ k3sup plan \
devices.json \
--user pi \
--servers 1 \
--background > bootstrap.sh
The output of that command is the [bootstrap.sh](http://bootstrap.sh)
file, which in this case looks like:
#!/bin/sh
echo "Setting up primary server 1"
k3sup install --host 192.168.2.2 \
--user pi \
--cluster \
--local-path kubeconfig \
--context picluster
echo "Fetching the server's node-token into memory"
export NODE_TOKEN=$(k3sup node-token --host 192.168.2.2 --user pi)
echo "Setting up worker: 1"
k3sup join \
--host 192.168.2.5 \
--server-host 192.168.2.2 \
--node-token "$NODE_TOKEN" \
--user pi &
echo "Setting up worker: 2"
k3sup join \
--host 192.168.2.6 \
--server-host 192.168.2.2 \
--node-token "$NODE_TOKEN" \
--user pi &
echo "Setting up worker: 3"
k3sup join \
--host 192.168.2.3 \
--server-host 192.168.2.2 \
--node-token "$NODE_TOKEN" \
--user pi &
echo "Setting up worker: 4"
k3sup join \
--host 192.168.2.4 \
--server-host 192.168.2.2 \
--node-token "$NODE_TOKEN" \
--user pi &
The output of running this command is a Kubeconfig file which allows us to access the cluster:
$ export KUBECONFIG=/path/to/kubeconfig
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
pi0 Ready control-plane,etcd,master 1m v1.28.7+k3s1
pi1 Ready worker 1m v1.28.5+k3s1
pi2 Ready worker 1m v1.28.5+k3s1
pi3 Ready worker 1m v1.28.5+k3s1
pi4 Ready worker 1m v1.28.5+k3s1
We will follow the installation instructions for configuring SpinKube with Helm on our cluster.
Install the CRDs required for SpinKube:
$ kubectl apply -f https://github.com/spinkube/spin-operator/releases/download/v0.1.0/spin-operator.crds.yaml
$ kubectl apply -f https://github.com/spinkube/spin-operator/releases/download/v0.1.0/spin-operator.runtime-class.yaml
$ kubectl apply -f https://github.com/spinkube/spin-operator/releases/download/v0.1.0/spin-operator.shim-executor.yaml
Install cert-manager:
# Cert manager CRDs
$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.3/cert-manager.crds.yaml
# Add and update Jetstack repository
helm repo add jetstack https://charts.jetstack.io
helm repo update
# Install the cert-manager Helm chart
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.14.3
Install the KWasm operator:
# Add Helm repository if not already done
helm repo add kwasm http://kwasm.sh/kwasm-operator/
# Install KWasm operator
helm install \
kwasm-operator kwasm/kwasm-operator \
--namespace kwasm \
--create-namespace \
--set kwasmOperator.installerImage=ghcr.io/spinkube/containerd-shim-spin/node-installer:v0.13.1
# Provision Nodes
kubectl annotate node --all kwasm.sh/kwasm-node=true
Install the Spin operator:
# Install Spin Operator with Helm
helm install spin-operator \
--namespace spin-operator \
--create-namespace \
--version 0.1.0 \
--wait \
oci://ghcr.io/spinkube/charts/spin-operator