Run nodelocal DNS on Rancher 2 custom cluster (RKE)

nodelocal-rancher.md

Run nodelocal DNS on Rancher 2 custom cluster (RKE)

WARNING: Nodelocal DNS is currently in alpha and these steps are for testing purposes only!

Due to issues with conntrack and DNS (See https://www.weave.works/blog/racy-conntrack-and-dns-lookup-timeouts and kubernetes/kubernetes#56903) K8s 1.13 added the nodelocal DNS option to avoid this condition. This steps help you install this nodelocal DNS option onto Rancher 2 custom cluster (RKE).

Setup kubectl or use embedded kubectl in UI

Make sure kubectl is installed and pointing to the cluster you want to change or use the embedded kubectl in the UI for your cluster.

https://kubernetes.io/docs/tasks/tools/install-kubectl/

kubectl get nodes
NAME            STATUS   ROLES                      AGE   VERSION
IP1             Ready    controlplane,etcd,worker   37m   v1.11.3
IP2             Ready    controlplane,etcd,worker   37m   v1.11.3
IP3             Ready    controlplane,etcd,worker   37m   v1.11.3

Install the nodelocal DNS daemon

This retrieves the definition from upstream:

• Replaces the DNS domain variable to cluster.local
• Gets the current DNS IP from kubectl (make sure your kubeconfig setup correctly)
• Uses 169.254.20.10 as local node IP for DNS requests (change this if this doesn't suit your environment)

curl -sL https://raw.githubusercontent.com/kubernetes/kubernetes/v1.13.0/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml | sed -e 's/__PILLAR__DNS__DOMAIN__/cluster.local/g' | sed -e "s/__PILLAR__DNS__SERVER__/$(kubectl get service --namespace kube-system kube-dns -o jsonpath='{.spec.clusterIP}')/g" | sed -e 's/__PILLAR__LOCAL__DNS__/169.254.20.10/g' | kubectl apply -f -

Validate it is running:

kubectl -n kube-system rollout status ds/node-local-dns
...
daemon set "node-local-dns" successfully rolled out

Validate it can be queried for DNS by logging into on of your cluster nodes using SSH and run the following command:

dig www.google.com @169.254.20.10 +short
216.58.206.68

Configure kubelet to point to the node local IP address

The local node IP address for DNS is configured as 169.254.20.10 in the previous command, we need to make sure the kubelet's point to this address as DNS IP.

Add the following to the cluster/RKE configuration (see https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/rke-clusters/options/#config-file for documentation):

services:
  kubelet:
    extra_args:
      cluster-dns: "169.254.20.10"

After saving the cluster configuration, Rancher will reprovision the cluster for you. If you are using RKE CLI, run rke up to apply the changes.

Validate that the configuration was applied by running the following command on one of your cluster nodes:

docker inspect kubelet | grep cluster-dns
            "--cluster-dns=169.254.20.10",
                "--cluster-dns=169.254.20.10",

Test DNS

Spin up an Ubuntu container and try to update apt sources.

> kubectl run -ti --rm --image=ubuntu ubuntu -- /bin/bash
If you don't see a command prompt, try pressing enter.
root@ubuntu-6fbcf757c7-4xsrq:/# apt update
Get:1 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]

Or try running nslookup:

kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup www.google.com