sebdeckers/README.md

k3s-playground ☸️

Goal: Discere faciendo. (Learn by doing.)

Mad scientists: @sakaru, @josephinekwa, @sebdeckers

Method:

Running the Dohnut container image from Docker Hub,
using k3s, a lightweight Kubernetes distribution,
on a stack of Raspberry Pi boards otherwise used for Tetris.

IPs are assigned by the LAN (wifi) DHCP server and may change at any time. These notes treat them as public IPs. In "production" an IP load balancer or DNS based solution (e.g. DDNS) would tie them together with an Internet-reachable address (IP or DNS).

Demo

Launch the app, triplicated, as a Deployment and Service described in dohnut.yaml.

$ k apply -f dohnut.yaml

A few moments later...

Verify that the app is running across the nodes.

$ k get -f dohnut.yaml
NAME             TYPE           CLUSTER-IP    EXTERNAL-IP                           PORT(S)        AGE
service/dohnut   LoadBalancer   10.43.1.233   10.17.0.105,10.17.0.192,10.17.0.195   53:32708/UDP   43m

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/dohnut   3/3     3            3           42m

$ k describe services dohnut
Name:                     dohnut
Namespace:                default
Labels:                   app=dohnut
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"dohnut"},"name":"dohnut","namespace":"default"},"spec":{...
Selector:                 app=dohnut
Type:                     LoadBalancer
IP:                       10.43.1.233
LoadBalancer Ingress:     10.17.0.105, 10.17.0.192, 10.17.0.195
Port:                     dns  53/UDP
TargetPort:               53/UDP
NodePort:                 dns  32708/UDP
Endpoints:                10.42.1.36:53,10.42.2.22:53,10.42.3.16:53
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

$ k get pods -o wide -l app=dohnut
NAME                      READY   STATUS    RESTARTS   AGE   IP           NODE             NOMINATED NODE   READINESS GATES
dohnut-76f957855f-5jrrg   1/1     Running   0          94m   10.42.1.36   k3s-server       <none>           <none>
dohnut-76f957855f-jkwgt   1/1     Running   0          94m   10.42.2.22   k3s-agent-blue   <none>           <none>
dohnut-76f957855f-xcgc5   1/1     Running   0          94m   10.42.3.16   k3s-agent-red    <none>           <none>

Sweet. Try using dig to send a DNS query to any of the nodes.

$ dig example.com @k3s-server
# ... or try the other nodes:
# $ dig example.com @k3s-agent-blue
# $ dig example.com @k3s-agent-red

; <<>> DiG 9.10.6 <<>> example.com @k3s-server
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30857
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.com.			IN	A

;; ANSWER SECTION:
example.com.		83182	IN	A	93.184.216.34

;; Query time: 67 msec
;; SERVER: 10.17.0.192#53(10.17.0.192)
;; WHEN: Wed Aug 28 16:07:19 +08 2019
;; MSG SIZE  rcvd: 56

Success! 🤩

How Does It Work?

As far as we can tell...

Each deployment is a separate instance of the container on one of the nodes.
The load balancer only exposes a public port. It does not actually route traffic between the nodes.

Setup

Each of the RPis were flashed with Raspbian Buster Lite (v4.19).

One RPi, k3s-server was set up using:

curl -sfL https://get.k3s.io | sh -

The other two RPis, k3s-agent-blue and k3s-agent-red, were set up to talk to the server as agents:

curl -sfL https://get.k3s.io | K3S_URL=https://k3s-server:6443 K3S_TOKEN=XXX sh -

(Where XXX is the value from /var/lib/rancher/k3s/server/node-token.)

`~/.bash_profile`

Consider setting up kubectl auto-completion for Bash/Zsh, if you're into that sort of thing.

alias k="kubectl"

`~/.ssh/config`

SSH access should only be needed for installation of k3s. Install kubectl on your local machine and use it to remotely manage the cluster.

One each of the nodes, as user pi (password raspberry):

curl https://github.com/sebdeckers.keys \
  >> ~/.ssh/authorized_keys
curl https://github.com/sakaru.keys \
  >> ~/.ssh/authorized_keys

Also a good idea to disable password login.

On local machine, for easier SSH access:

Host k3s-agent-blue
  User pi
  Hostname 10.17.0.105

Host k3s-agent-red
  User pi
  Hostname 10.17.0.195

Host k3s-server
  User pi
  Hostname 10.17.0.192

`/etc/hosts`

Set up some poor man's DNS records on all of the machines as well as local machine:

10.17.0.105 k3s-agent-blue
10.17.0.195 k3s-agent-red
10.17.0.192 k3s-server

`~/.kube/config`

Copy the generated config with credentials from server to the local machine. Edit the connection URL:

server: https://k3s-server:6443

See: k3s Documentation: Accessing Cluster from Outside

	---
	apiVersion: v1
	kind: Service
	metadata:
	name: dohnut
	labels:
	app: dohnut
	spec:
	selector:
	app: dohnut
	type: LoadBalancer
	ports:
	- name: dns
	protocol: UDP
	port: 53 # External
	targetPort: 53 # Docker
	---
	apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: dohnut
	labels:
	app: dohnut
	spec:
	replicas: 3
	selector:
	matchLabels:
	app: dohnut
	template:
	metadata:
	labels:
	app: dohnut
	spec:
	containers:
	- name: dohnut
	image: commonshost/dohnut
	# command: ["dohnut"]
	args: ['--listen', '0.0.0.0:53', '--doh', 'commonshost', '--bootstrap', '1.1.1.1']
	# ports:
	# - containerPort: 80
	---