Configuring Microk8s for remote access

In order to learn about Kubernetes, I've installed Microk8s on Ubuntu Server on an old laptop stashed in my basement. It seemed like a better option than Minikube because Microk8s actually claims to have auto-update. (I really want auto-update because I'm a developer - not ops. I set things up, but I'm bad at maintaining them day over day.)

I got it installed okay, but I ran into trouble connecting to it remotely. In my opinion, the documentation for doing this is SUPER CONFUSING for beginners. You have to spend way more time than you expect learning what these things are.

Out-of-the-box, Microk8s appears to be setup to authenticate with a "bootstrap token". This auth token only works when you're connecting locally. But I don't want to connect locally, I want to:

Connect from my laptop upstairs to manage remotely manually
Connect from a pipeline worker (incidentally but not necessarily running on the same box) so I can manage the infrastructure as code.

Doing so throws me deeeeeeep into a rabbit hole of Kubernetes authentication with zero protection.

First problem: Kubernetes has zero concept of "user authentication":

In this regard, Kubernetes does not have objects which represent normal user accounts. Normal users cannot be added to a cluster through an API call.

The docs offer a couple alternatives:

The Options

Setup an X509 client certificate
- The docs strongly recommend you take this option, so I did too.
- That said, it's its own special brand of hell if you aren't intimately familiar with certificates and running openssl on the command line (which I'm only barely!)
- I think this is comparable to an SSH keypair. But worse. I miss SSH keypairs here.
Static Token File
- You create a file somewhere that contains a bunch of tokens
- This looks super simple
- BUT: I've got no idea how to configure Microk8s to do this. Like, you have to modify the command line options Kubernetes starts up with, but I'm not using real Kubernetes. Microk8s has a file containing CLI parameters, but when I tried to edit that file, Microk8s failed to start up (and I don't know where the logs went). So I'm scared of editing that file.
Bootstrap Tokens
- I think this is just what Microk8s gives you to start out. They won't work for us for the reason I described earlier.
Service Account Tokens
- I don't know what these are yet. I think I'll need them when I setup pipeline access. But I'm just concentrating on my own remote access first.
OpenID Connect Tokens
- Super shiny. I'd use this, except I'd want to combine it with my own KeyCloak instance. But I want to run KeyCloak ON TOP OF Kubernetes, and I can't rely on a circular dependency. If KeyCloak goes down, I need to be able to connect to Kubernetes to stand it back up.
- I might end up adding this eventually, but it's not the main solution.
- I guess I could connect it to Google Auth instead, but the reason I'm planning to run my own KeyCloak is because I'm dissatisfied with the token lifetimes it provides.
- One more thing: It's not clear to me how to add this token to kubectl in my terminal. I think I'd probably need to build my own client to "log in" and get a bearer token. That sounds like unnecessary work. If I was building a webapp frontend for Kubernetes then this would be appropriate. But not a terminal.
Webhook Token Authentication
- I assume this requires a separate API to be running. Can't do this for the same reason I can't do OpenID Connect with my own KeyCloak instance.
Authenticating Proxy
- A reverse proxy server in front of Kubernetes checks the authentication
- No, I don't have this and I don't plan to make one.

Quit stalling and get on with it

My Setup

Ubuntu 22.04 LTS running on a 1.1 GHz single-core (w/ hyperthreading) Intel Atom EeePC netbook
- I'm pretty sure they sell Raspberry Pis that are more powerful than this. But at least I'm not mixing processor architecture families.
- My home router is configured to give this a fixed IP address (192.168.███.███). This would be impossible to connect to otherwise.
- I can connect to this by hostname on my LAN through NetBIOS ($HOSTNAME.local) because I think I enabled that feature on Samba at some point.
- I'm not planning to expose this to the public internet through routing. I'm hoping to use a Cloudflare tunnel instead.
A Windows 10 laptop

Certificates!

The general outline is:

Create a key
Creates a Certificate Signing Request (CSR)
From the Microk8s server, wrap the CSR in a Kubernetes resource file and send it to Kubernetes.
- This request includes the key from earlier
Approve the request
Download the certificate
Create the Role and RoleBinding
Copy the cert to your own computer
Configure kubectl to use it

This is all based on this documentation page: https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#normal-user

Okay, outline out of the way, how does all this work?

1. Create a key

# the docs use "myuser". I've replaced this with $USER because nobody calls
# themselves "myuser".
openssl genrsa -out $USER.key 2048

This command creates a private key file. It's the least complicated part of this process.

2. Create a Certificate Signing Request

"Certificate Signing Requests" are an arcane thing to me. Based on what I've seen on /r/sysadmin I think they're arcane to basically everybody.

It's extra weird here because normally certificates are used to authenticate servers, e.g. https://google.com has a certificate. But here, we need a cert to authenticate a person.

openssl req -new -key myuser.key -out myuser.csr

The command asks a series of questions for information that'll ultimately get embedded inside the certificate. Annoyingly, there doesn't seem to be any way to pass responses via command line parameters, and I don't know why. It might have to do with some random config file (openssl.cnf) living in your filesystem deciding what questions you should be asked.

IMO the docs do NOT give enough information on how to answer these questions. I mean, it gives you a bit of advice, but the questions don't match up.

It is important to set CN and O attribute of the CSR. CN is the name of the user and O is the group that this user will belong to.

CN is short for "Common Name". In this case it's your username. It is NOT the FQDN of your Microk8s server. (Found that out the hard way.)
O is short for "Organization". Normally it's a company name, but the docs say Kubernetes reappropriates this field to refer to refer to a group name. I put my name in this field and I haven't run into a problem yet.

This was more or less my response:

$ openssl req -new -key $USER.key -out $USER.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Your-State-Here
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:groupnamehereithink
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:usernamehere
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

3. Create the CSR object and send to Kubernetes

Like everything else in Kubernetes, you create objects by building a YAML file and feeding it to kubectl apply.

The docs recommend an expiration of one day (86400 seconds). That means the certificate you worked so hard for will expire tomorrow.

So unless you plan on running this dang thing every single day, I recommend increasing that. I've changed it to 4 years below.

cat <<EOF | microk8s kubectl apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: $USER
spec:
  request: $(cat $USER.csr | base64 | tr -d "\n")
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: $((60*60*24*365*4))
  usages:
  - client auth
EOF

4. Approve the CSR

Make sure the CSR exists:

$ microk8s kubectl get csr
NAME           AGE   SIGNERNAME                            REQUESTOR   REQUESTEDDURATION   CONDITION
usernamehere   95s   kubernetes.io/kube-apiserver-client   admin       4y                  Pending

Approve it.

$ microk8s kubectl certificate approve $USER
certificatesigningrequest.certificates.k8s.io/usernamehere approved

5. Download the certificate

A bit of background: You can view a representation of the CSR like this: (We don't need it -- it's comforting to see it though)

microk8s kubectl get csr/$USER -o yaml

We don't need this whole thing. We only need the certificate. The docs say we can download and extract it like this:

microk8s kubectl get csr $USER -o jsonpath='{.status.certificate}'| base64 -d > $USER.crt

6. Create the Role and Rolebinding

The "user" exists, but it can't do anything yet. We need to give it permissions.

The docs say it's a bad idea to give accounts wildcard permissions. But we're the only user of this goshdarn thing.

microk8s kubectl create role developer --verb='*' --resource='*'
microk8s kubectl create rolebinding developer-binding-$USER --role=developer --user=$USER

7. Copy the cert to your own computer

Somehow, copy $USER.crt and $USER.key to your own machine. The method doesn't matter. You could do this with scp. I've been using Visual Studio remote SSH to download them.

The $USER.key file could be catted and copy/pasted. but the $USER.crt file is binary.

8. Configure kubectl to use the cert

This bit's probably the least clean of these instructions because I had this whole thing "almost working" several times and then had to tweak something at the last minute.

The docs say to run this command: (*I'm using $USERNAME here because I'm running this in Git Bash on Windows, and $USERNAME is the name of the env var. My laptop happens to use the same username as my server!)

kubectl config set-credentials $USERNAME --client-key=$USERNAME.key --client-certificate=$USERNAME.crt --embed-certs=true

I think that should add a section to your ~/.kube/config file like this:

users:
- name: usernamehere
  user:
    client-certificate-data: █████████████==
    client-key-data: ██████████████

Next I think we need to add a "cluster" definition for this remote server. For this, we jump over to this documentation page.

I used the IP address of your Microk8s even though NetBIOS gives me a domain name because something between Microk8s and kubectl refuses to allow it. Microk8s uses a config file (/var/snap/microk8s/current/certs/csr.conf.template) that contains default settings for the server certificate, but when I try to change that file, Microk8s reverts the changes! A bunch of people in this GitHub issue struggle with having rotating IP addresses and being unable to edit this file. The alternative is to pass --insecure-skip-tls-verify every time you run kubectl.

Basically, static IPs are the way to go here.

kubectl config set-cluster microk8s-cluster --server=https://192.168.███.███:16443

clusters:
- cluster:
    certificate-authority-data: █████████████==
    server: https://192.168.███.███:16443
  name: microk8s-cluster

Let's create a "context" to associate the user and server:

kubectl config set-context microk8s --cluster=microk8s-cluster --user=$USERNAME

Finally, let's set that as the active context. I'd previously installed Rancher Desktop so I had Kubectl, but it was pointing to my local install of Kubernetes instead of my basement server.

kubectl config use-context microk8s

Addendum: Error `x509: certificate signed by unknown authority`

I moved servers and tried following my own instructions, except instead of manually generating the key and CSR I copied those files from the old machine. Needless to say, I followed my instructions haphazardly.

(I'm not going to go into the kerfuffle of needing to setup my local Kube config with TWO different "users" (each with their own certificate) which meant adding a -${servername} suffix to one of them.)

Anyways, when I tried to run a Kubernetes command I got this weird error:

I0308 08:47:08.770403   11380 versioner.go:58] Get "https://192.168.███.███:16443/version?timeout=5s": x509: certificate signed by unknown authority
E0308 08:47:08.964606   23040 memcache.go:238] couldn't get current server API group list: Get "https://192.168.███.███:16443/api?timeout=32s": x509: certificate signed by unknown authority
E0308 08:47:08.992410   23040 memcache.go:238] couldn't get current server API group list: Get "https://192.168.███.███:16443/api?timeout=32s": x509: certificate signed by unknown authority
E0308 08:47:09.008620   23040 memcache.go:238] couldn't get current server API group list: Get "https://192.168.███.███:16443/api?timeout=32s": x509: certificate signed by unknown authority
E0308 08:47:09.034709   23040 memcache.go:238] couldn't get current server API group list: Get "https://192.168.███.███:16443/api?timeout=32s": x509: certificate signed by unknown authority
E0308 08:47:09.059874   23040 memcache.go:238] couldn't get current server API group list: Get "https://192.168.███.███:16443/api?timeout=32s": x509: certificate signed by unknown authority
Unable to connect to the server: x509: certificate signed by unknown authority

This was because my clusters[].cluster entry in ~/.kube/config needed an key called certificate-authority-data. I don't remember which step was supposed to create this, and I wasn't going to start all over from scratch if I could help it.

This StackOverflow answer was the most helpful.

It had me run this command:

openssl s_client -showcerts -connect 192.168.███.███:16443

I grabbed this chunk of text and put it into a text file: (INCLUDING the header/footer!)

At this point I examined my other cluster's certificate-authority-data key to see what the heck kind of format it expected. I discovered that if I base64-decoded it, it was a chunk of Base64 data that was wrapped with the ----BEGIN/END CERTIFICATE---- lines -- exactly what I just copied. So now I just need to do the reverse operation on the text I just copied:

cat copiedtext.txt | base64

Then I added the base64 text to ~/.kube/config:

clusters:
- cluster:
    certificate-authority-data: ███████████████████...████████████████████████████████████████████
    server: https://192.168.███.███:16443
  name: second-cluster
  #...

After this, I was able to run kubectl commands successfully again.

karlrwjohnson/Connecting to Microk8s remotely.md