Raspberry Pi Cluster with ClickHouse and K3s Setup Steps

picluster.md

Step 1: Setup the Raspberry Pis

1. Install Raspbian Lite on an SD Card
2. Boot the Pi
3. Copy the disk to the NVMe:

sudo dd if=/dev/mm** of=/dev/nvme** bs=32M

1. Shutdown, remove the SD Card, and reboot
2. Change the pi hostname to something unique
3. Expand the filesystem:

sudo raspi-config --expand-rootfs

1. edit cgroup settings in /boot/firmware/cmdline.txt:

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

1. Update the pi firmware to apply the above setttings:

sudo rpi-update

1. Reboot the Pi

Step 2: Install K3s

1. Install the control-plane node on pi1:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" sh -s - --flannel-backend none

1. Setup pi2 and pi3 as worker nodes (get K3S_TOKEN from /var/lib/rancher/k3s/server/node-token on pi1):

curl -sfL https://get.k3s.io | K3S_URL=https://pi1:6443 K3S_TOKEN=[YOUR TOKEN] sh -s -

1. Copy the k3s.yaml and export it as your kubeconfig:

scp user@pi1:/etc/rancher/k3s/k3s.yaml ./k3s.yaml

export KUBECONFIG=./k3s.yaml

1. Install cilium as a CNI:

helm upgrade cilium cilium/cilium --version 1.16.5 \
   --namespace kube-system \
   --reuse-values \
   --set l2announcements.enabled=true \
   --set k8sClientRateLimit.qps=10 \
   --set k8sClientRateLimit.burst=60 \
   --set kubeProxyReplacement=true \
   --set k8sServiceHost=pi1 \
   --set k8sServicePort=6443

Step 3: Install the Altinity Operator

kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

Step 4: Create a replicated ClickHouse Cluster

helm repo add altiniy https://helm.altinity.com

helm install clickhouse-dev --create-namespace --namespace clickhouse altinity/clickhouse  --set keeper.enabled=true --set clickhouse.replicasCount=2

Test Queries:

SELECT * FROM system.zookeeper WHERE path = '/'

CREATE TABLE IF NOT EXISTS test_rep ON CLUSTER `{cluster}`
(
    `number` UInt32,
    `created_at` DateTime DEFAULT now()
)
ENGINE = ReplicatedMergeTree
ORDER BY number;

INSERT INTO test_rep (number) SELECT number
FROM system.numbers
LIMIT 10;

SELECT hostName(), *
FROM clusterAllReplicas('{cluster}', default.test_rep)
ORDER BY 1 ASC, 2 ASC