Kubernetes on RaspberryPi ClusterHAT

README.md

Abandoned, Kubernetes, although possible to compile to run on a pi zero, was running into other issues when it came to enrolling the Zero into the cluster. Benched this to work on Docker Swarm for now which is much easier to setup.

How to get Kubernetes on a RaspberryPi ClusterHAT

This is a basic guide to getting Kubernetes running on a RaspberryPi ClusterHAT.

Hardware Used:

• 1x RaspberryPi 3 B+
• 4x RaspberryPi Zero
• 1x ClusterHAT
• 4x 16GB Class 10 SD Card (Zeros) && 1x 32GB Class 10 SD Card (RaspberryPI 3)

Download the images and flash them onto the SD Cards

1. Visit https://clusterhat.com/setup-software and download the Lite-Combined zip package (contains controller & 4 pi images)
2. Flash these onto SD cards and insert them into the appropriate RaspberryPi
   • Ensure to touch ssh in the boot volume of the SD card to ensure you can SSH onto the Pis after flashing.
3. Attach RaspberryPi Zeros into appropriate slots on the ClusterHAT
4. Plug RaspberryPi 3 into power

Intial Setup and Securing the Pis (This should be done on all Pis)

1. Start all your Pis clusterhat on, They will all be assigned IPs which you can find the same way you found the controller IP.
2. SSH onto the controller, you should be able to find the IP address by looking at your router for connected devices, or other means.
   • ssh pi@<IP ADDRESS> username/password by default is pi/clusterhat
3. Change the password from the default passwd to something better

The next steps are optional, but highly recommended

4. Add your public key to ~/.ssh/authorized_keys to enable passwordless login
   • Ensure ~/.ssh permissions are 700 chmod 700 ~/.ssh
   • Ensure ~/.ssh/authorized_keys permissions are 600 chmod 600 ~/.ssh/authorized_keys
5. Logout of your current SSH session, and relogin in to ensure it doesn't ask your password
6. Disable password authentication for SSH

• sudo vi /etc/ssh/sshd_config and uncomment PasswordAuthentication yes and change it to PasswordAuthentication no

7. Restart SSH service sudo service ssh restart

Controller Setup

1. Run curl -sSL https://get.docker.com | sh which will automatically install docker on your host

NOTE After installing Docker you will lose access to your Zeros. Add the following iptables rules to the first section in /etc/iptables/rules.v4

• -P FORWARD ACCEPT

which requires you to readd this rule everytime the controller is restarted, or add the following rules to /etc/iptables/rules.v4

• -A FORWARD -s 192.168.1.0/24 -j ACCEPT
• -A FORWARD -d 192.168.1.0/24 -j ACCEPT

change the CIDR to match your LAN. These rules will persist on reboot.

Install kubeadm

1. curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
2. echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
3. apt-get update && apt-get install -y kubeadm

Setup Kubernetes

1. Append cgroup_enable=memory cgroup_memory=1 to /boot/cmdline.txt and sudo update-rc.d dphys-swapfile remove then reboot
2. kubeadm init --pod-network-cidr 10.244.0.0/16

Setup the Zeros

Install docker-ce

Add the following to /etc/apt/prteferences.d/docker-ce since the latest docker does not work on pi zeros at the moment.

Package: docker-ce
Pin: version 18.06.*
Pin-Priority: 1000

1. Then Run curl -sSL https://get.docker.com | sh which will automatically install docker on your host

Compile Kubernetes For arm32v5

Unfortunately Kubernetes stopped supported arm32v5 from version 1.6 onward, so we need to compile it ourselves.

Fortunately we can cross compile it, so on your windows (use windows subsystem for linux [WSL])/mac/linux box install go1.11.1+ and clone the Kubernetes repo https://github.com/kubernetes/kubernetes

1. Apply the following diff to the Kubernetes repo

diff --git a/hack/lib/golang.sh b/hack/lib/golang.sh
index 41004ced12..906b3465ae 100755
--- a/hack/lib/golang.sh
+++ b/hack/lib/golang.sh
@@ -308,6 +308,7 @@ kube::golang::set_platform_envs() {

   export GOOS=${platform%/*}
   export GOARCH=${platform##*/}
+  export GOARM=5

   # Do not set CC when building natively on a platform, only if cross-compiling from linux/amd64
   if [[ $(kube::golang::host_platform) == "linux/amd64" ]]; then
@@ -316,7 +317,7 @@ kube::golang::set_platform_envs() {
     case "${platform}" in
       "linux/arm")
         export CGO_ENABLED=1
-        export CC=arm-linux-gnueabihf-gcc
+        export CC=arm-linux-gnueabi-gcc
         ;;
       "linux/arm64")
         export CGO_ENABLED=1

2 install required deps apt install gcc-arm-linux-gnueabi

Next we make each cmd that is required to run on a node.

3. make all WHAT=cmd/kubeadm KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm
4. make all WHAT=cmd/kubelet KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm
5. make all WHAT=cmd/kubectl KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm
6. make all WHAT=cmd/kube-proxy KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm
7. zip up the cmds and scp them to each Zero node you have.
   • tar -zcvf cmds.tar.gz _output/local/bin/linux/arm/
   • scp cmds.tar.gz [email protected]:~
8. untar on Zero tar -xzf cmds.tar.gz mv _output/local/bin/linux/arm/* .
9. Ensure the binaries work by checking the version

> ./kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.0-alpha.3.183+49e83f899807d4-dirty", GitCommit:"49e83f899807d41a87065fde15822ac1c070891a", GitTreeState:"dirty", BuildDate:"2019-02-15T05:28:42Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/arm"}

10. Move them all to /usr/bin

Join the master Kubernetes node

1. Add cgroup_enable=cpuset group_enable=memory cgroup_memory=1
2. Create a file /etc/systemd/system/kubelet.service with the following contents

[Unit]
Description=kubelet

[Service]
Restart=always
ExecStart=/usr/bin/kubelet \
  --allow-privileged=true \
  --enable-server=false \
  --hostname-override=127.0.0.1 \
  --config=/etc/kubernetes/config.yaml \
  --pod-infra-container-image="k8s.gcr.io/pause-arm:3.1" \
  -v2

[Install]
WantedBy=multi-user.target

12. The the following commands to enable the service
    • sudo systemctl daemon-reload
    • sudo systemctl enable kubelet
    • sudo systemctl start kubelet
13. Ensure it is running sudo systemctl status kubelet