Abandoned, Kubernetes, although possible to compile to run on a pi zero, was running into other issues when it came to enrolling the Zero into the cluster. Benched this to work on Docker Swarm for now which is much easier to setup.

How to get Kubernetes on a RaspberryPi ClusterHAT

This is a basic guide to getting Kubernetes running on a RaspberryPi ClusterHAT.

Hardware Used:

1x RaspberryPi 3 B+
4x RaspberryPi Zero
1x ClusterHAT
4x 16GB Class 10 SD Card (Zeros) && 1x 32GB Class 10 SD Card (RaspberryPI 3)

Download the images and flash them onto the SD Cards

Visit https://clusterhat.com/setup-software and download the Lite-Combined zip package (contains controller & 4 pi images)
Flash these onto SD cards and insert them into the appropriate RaspberryPi
- Ensure to touch ssh in the boot volume of the SD card to ensure you can SSH onto the Pis after flashing.
Attach RaspberryPi Zeros into appropriate slots on the ClusterHAT
Plug RaspberryPi 3 into power

Intial Setup and Securing the Pis (This should be done on all Pis)

Start all your Pis clusterhat on, They will all be assigned IPs which you can find the same way you found the controller IP.
SSH onto the controller, you should be able to find the IP address by looking at your router for connected devices, or other means.
- ssh pi@<IP ADDRESS> username/password by default is pi/clusterhat
Change the password from the default passwd to something better

The next steps are optional, but highly recommended

Add your public key to ~/.ssh/authorized_keys to enable passwordless login
- Ensure ~/.ssh permissions are 700 chmod 700 ~/.ssh
- Ensure ~/.ssh/authorized_keys permissions are 600 chmod 600 ~/.ssh/authorized_keys
Logout of your current SSH session, and relogin in to ensure it doesn't ask your password
Disable password authentication for SSH

sudo vi /etc/ssh/sshd_config and uncomment PasswordAuthentication yes and change it to PasswordAuthentication no

Restart SSH service sudo service ssh restart

Controller Setup

Run curl -sSL https://get.docker.com | sh which will automatically install docker on your host

NOTE After installing Docker you will lose access to your Zeros. Add the following iptables rules to the first section in /etc/iptables/rules.v4

-P FORWARD ACCEPT

which requires you to readd this rule everytime the controller is restarted, or add the following rules to /etc/iptables/rules.v4

-A FORWARD -s 192.168.1.0/24 -j ACCEPT
-A FORWARD -d 192.168.1.0/24 -j ACCEPT

change the CIDR to match your LAN. These rules will persist on reboot.

Install kubeadm

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
echo "deb http://apt.kubernetes.io/ kubernetes-xenial main" > /etc/apt/sources.list.d/kubernetes.list
apt-get update && apt-get install -y kubeadm

Setup Kubernetes

Append cgroup_enable=memory cgroup_memory=1 to /boot/cmdline.txt and sudo update-rc.d dphys-swapfile remove then reboot
kubeadm init --pod-network-cidr 10.244.0.0/16

Setup the Zeros

Install docker-ce

Add the following to /etc/apt/prteferences.d/docker-ce since the latest docker does not work on pi zeros at the moment.

Package: docker-ce
Pin: version 18.06.*
Pin-Priority: 1000

Then Run curl -sSL https://get.docker.com | sh which will automatically install docker on your host

Compile Kubernetes For arm32v5

Unfortunately Kubernetes stopped supported arm32v5 from version 1.6 onward, so we need to compile it ourselves.

Fortunately we can cross compile it, so on your windows (use windows subsystem for linux [WSL])/mac/linux box install go1.11.1+ and clone the Kubernetes repo https://github.com/kubernetes/kubernetes

Apply the following diff to the Kubernetes repo

diff --git a/hack/lib/golang.sh b/hack/lib/golang.sh
index 41004ced12..906b3465ae 100755
--- a/hack/lib/golang.sh
+++ b/hack/lib/golang.sh
@@ -308,6 +308,7 @@ kube::golang::set_platform_envs() {

   export GOOS=${platform%/*}
   export GOARCH=${platform##*/}
+  export GOARM=5

   # Do not set CC when building natively on a platform, only if cross-compiling from linux/amd64
   if [[ $(kube::golang::host_platform) == "linux/amd64" ]]; then
@@ -316,7 +317,7 @@ kube::golang::set_platform_envs() {
     case "${platform}" in
       "linux/arm")
         export CGO_ENABLED=1
-        export CC=arm-linux-gnueabihf-gcc
+        export CC=arm-linux-gnueabi-gcc
         ;;
       "linux/arm64")
         export CGO_ENABLED=1

2 install required deps apt install gcc-arm-linux-gnueabi

Next we make each cmd that is required to run on a node.

make all WHAT=cmd/kubeadm KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm
make all WHAT=cmd/kubelet KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm
make all WHAT=cmd/kubectl KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm
make all WHAT=cmd/kube-proxy KUBE_VERBOSE=5 KUBE_BUILD_PLATFORMS=linux/arm
zip up the cmds and scp them to each Zero node you have.
- tar -zcvf cmds.tar.gz _output/local/bin/linux/arm/
- scp cmds.tar.gz [email protected]:~
untar on Zero tar -xzf cmds.tar.gz mv _output/local/bin/linux/arm/* .
Ensure the binaries work by checking the version

> ./kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.0-alpha.3.183+49e83f899807d4-dirty", GitCommit:"49e83f899807d41a87065fde15822ac1c070891a", GitTreeState:"dirty", BuildDate:"2019-02-15T05:28:42Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/arm"}

Move them all to /usr/bin

Join the master Kubernetes node

Add cgroup_enable=cpuset group_enable=memory cgroup_memory=1
Create a file /etc/systemd/system/kubelet.service with the following contents

[Unit]
Description=kubelet

[Service]
Restart=always
ExecStart=/usr/bin/kubelet \
  --allow-privileged=true \
  --enable-server=false \
  --hostname-override=127.0.0.1 \
  --config=/etc/kubernetes/config.yaml \
  --pod-infra-container-image="k8s.gcr.io/pause-arm:3.1" \
  -v2

[Install]
WantedBy=multi-user.target

The the following commands to enable the service
- sudo systemctl daemon-reload
- sudo systemctl enable kubelet
- sudo systemctl start kubelet
Ensure it is running sudo systemctl status kubelet

mchestr/README.md