Cilium on K3s / vSphere

Pre-requisites

In this example I’ve three ‘controlplane’ nodes created as control{0..2} and five ‘worker’ nodes worker{0..5} created running openSUSE Leap 15.2.

All IP addresses assigned using DHCP, I’m using IP lookups via govc (https://github.com/vmware/govmomi/tree/master/govc#govc) as a poor man’s DNS.

Finally, I’m using k3sup (https://github.com/alexellis/k3sup) to bootstrap the installation.

Install the first node:

export K3S_VERSION="v1.19.8+k3s1"
export K3S_OPTIONS="--node-taint CriticalAddonsOnly=true:NoExecute --flannel-backend=none --no-flannel"

k3sup install --cluster --ip $(govc vm.ip /int/vm/control0) \
  --user nick --local-path ~/.kube/cilium.yaml --context cilium \
  --k3s-version $K3S_VERSION --k3s-extra-args $K3S_OPTIONS

Install the second and third control plane nodes:

k3sup join --ip $(govc vm.ip /int/vm/control1) \
  --server-ip $(govc vm.ip /int/vm/control0) \
  --server --server-user nick --user nick \
  --k3s-version $K3S_VERSION --k3s-extra-args $K3S_OPTIONS

k3sup join --ip $(govc vm.ip /int/vm/control2) \
  --server-ip $(govc vm.ip /int/vm/control0) \
  --server --server-user nick --user nick \
  --k3s-version $K3S_VERSION --k3s-extra-args $K3S_OPTIONS

Install worker nodes:

for worker in $(for node in $(govc ls /int/vm | grep worker) ; do govc vm.ip $node ; done) ; do echo $worker ; done \
 | parallel -v -I% k3sup join --ip % --server-ip $(govc vm.ip /int/vm/control0) \
 --user nick --k3s-version $K3S_VERSION

Install Cilium and Hubble:

kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/quick-install.yaml
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/quick-hubble-install.yaml

Generate connectivity tests:

kubectl create ns cilium-test
kubectl apply -n cilium-test -f https://raw.githubusercontent.com/cilium/cilium/v1.9/examples/kubernetes/connectivity-check/connectivity-check.yaml

Connect to Hubble UI via http://localhost:12000:

kubectl port-forward -n kube-system svc/hubble-ui --address 0.0.0.0 --address :: 12000:80

yankcrime/cilium-k3s.org

Cilium on K3s / vSphere

Pre-requisites