Skip to content

Instantly share code, notes, and snippets.

@mattfenwick

mattfenwick/dns.md

Last active August 10, 2020 11:38
Show Gist options
  • Save mattfenwick/f3c38b570dbd7970c1e377197169409e to your computer and use it in GitHub Desktop.
Save mattfenwick/f3c38b570dbd7970c1e377197169409e to your computer and use it in GitHub Desktop.
Download ZIP
using node-local-dns in kube
Raw
dns.md

Setting up local registry:

for node in $(kind get nodes --name $CLUSTER_NAME); do
  kubectl annotate --overwrite node "${node}" "kind.x-k8s.io/registry=localhost:${reg_port}";
done

In kubernetes/dns repo:

make build && make containers

docker tag gcr.io/k8s-staging-dns/k8s-dns-node-cache-amd64:1.15.13-6-g5596a13-dirty localhost:5000/k8s-dns-node-cache:latest

docker push localhost:5000/k8s-dns-node-cache:latest 

# change image to localhost:5000/k8s-dns-node-cache:latest and pullPolicy to Always
kubectl edit ds -n kube-system node-local-dns

# grant privileges to node-local-dns to be able to pull nodes
kubectl create clusterrolebinding node-local-dns-crb --clusterrole=cluster-admin --serviceaccount=kube-system:node-local-dns
Raw
kind-create-ipv6.sh
IMAGE=kindest/node:latest
kind build node-image --image $IMAGE
reg_name='kind-registry'
reg_port='5000'
CLUSTER_NAME=kind-ipv6
cat <<EOF | kind create cluster --name $CLUSTER_NAME --image $IMAGE --config=-
# an ipv6 cluster
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
containerdConfigPatches:
- |-
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:${reg_port}"]
endpoint = ["http://${reg_name}:${reg_port}"]
networking:
# disableDefaultCNI: true
ipFamily: ipv6
apiServerAddress: 127.0.0.1
EOF
Raw
kind-hacking.md

Run on mac:

git clone [email protected]:kubernetes/kubernetes

cd kubernetes

make kubeadm

sudo ./_output/local/go/bin/kubeadm init --v=5 --ignore-preflight-errors=KubeletVersion,SystemVerification

If following the kind setup instructions:

git clone [email protected]:kubernetes/kubernetes $(go env GOPATH)/src/k8s.io/kubernetes

cd $(go env GOPATH)/src/k8s.io/kubernetes

Following these instructions To create a cluster from Kubernetes source:

brew install gnu-tar 

kind build node-image --image $IMAGE

kind create cluster --image $IMAGE

Kind hacking

Run from a version of kubernetes other than master:

cd ~/path/to/kubernetes
git checkout --track remotes/origin/release-1.17

kind build node-image --image=k8s:1.17 

CLUSTER_NAME=my-kind-cluster
kind -v 10 create cluster --image k8s:1.17 --name $CLUSTER_NAME

kind get nodes --name $CLUSTER_NAME

docker inspect ${CLUSTER_NAME}-control-plane

Look at what's going on inside the control plane:

docker exec -ti ${CLUSTER_NAME}-control-plane bash

# the rest of these commands are run from inside the docker container!

journalctl -flu kubelet

more /etc/kubernetes/manifests/etcd.yaml

crictl ps

more /kind/manifests/default-cni.yaml
more /kind/manifests/default-storage.yaml

# don't run these next two unless you know what you're doing!  this will blow away your previous credentials
# kubeadm reset -f
# kubeadm init --config /kind/kubeadm.conf

more /etc/kubernetes/admin.conf

ls /var/log/containers/

Look at what's going on in Docker:

docker ps
docker images
docker container ls
Raw
kind-verbosity.md

Use docker ps or docker container list to find the container id of one of your KinD nodes:

✗ docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                       NAMES
defc10620c4b        kindest/node:latest   "/usr/local/bin/entr…"   3 hours ago         Up 3 hours          127.0.0.1:64168->6443/tcp   kind-lat-control-plane
67fdaa647877        registry:2            "/entrypoint.sh /etc…"   2 weeks ago         Up 9 days           0.0.0.0:5000->5000/tcp      kind-registry

Exec into that container:

docker exec -ti defc10620c4b bash

Inside of the container:

apt-get update -y && apt-get install -y vim

vim /var/lib/kubelet/kubeadm-flags.env

# add ' --v=10' into the command string in /var/lib/kubelet/kubeadm-flags.env

systemctl restart kubelet.service

journalctl -flu kubelet
Raw
node_local_dns_setup.md

https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/

grab this file

run these commands:

kubedns=$(kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP})
localdns=169.254.20.10
domain=cluster.local

# Mac os is weird: it needs the extra '' after the -i flag
sed -i '' "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml

kubectl create -f nodelocaldns.yaml

Transformed yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns-upstream
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "KubeDNSUpstream"
spec:
  ports:
    - name: dns
      port: 53
      protocol: UDP
      targetPort: 53
    - name: dns-tcp
      port: 53
      protocol: TCP
      targetPort: 53
  selector:
    k8s-app: kube-dns
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  Corefile: |
    cluster.local:53 {
        errors
        cache {
                success 9984 30
                denial 9984 5
        }
        reload
        loop
        bind 169.254.20.10 10.96.0.10
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        health 169.254.20.10:8080
        }
    in-addr.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10 10.96.0.10
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        }
    ip6.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10 10.96.0.10
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        }
    .:53 {
        errors
        cache 30
        reload
        loop
        bind 169.254.20.10 10.96.0.10
        forward . __PILLAR__UPSTREAM__SERVERS__ {
                force_tcp
        }
        prometheus :9253
        }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    k8s-app: node-local-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    metadata:
      labels:
        k8s-app: node-local-dns
      annotations:
        prometheus.io/port: "9253"
        prometheus.io/scrape: "true"
    spec:
      priorityClassName: system-node-critical
      serviceAccountName: node-local-dns
      hostNetwork: true
      dnsPolicy: Default  # Don't use cluster DNS.
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
        - effect: "NoExecute"
          operator: "Exists"
        - effect: "NoSchedule"
          operator: "Exists"
      containers:
        - name: node-cache
          image: k8s.gcr.io/k8s-dns-node-cache:1.15.13
          resources:
            requests:
              cpu: 25m
              memory: 5Mi
          args: [ "-localip", "169.254.20.10,10.96.0.10", "-conf", "/etc/Corefile", "-upstreamsvc", "kube-dns-upstream" ]
          securityContext:
            privileged: true
          ports:
            - containerPort: 53
              name: dns
              protocol: UDP
            - containerPort: 53
              name: dns-tcp
              protocol: TCP
            - containerPort: 9253
              name: metrics
              protocol: TCP
          livenessProbe:
            httpGet:
              host: 169.254.20.10
              path: /health
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
          volumeMounts:
            - mountPath: /run/xtables.lock
              name: xtables-lock
              readOnly: false
            - name: config-volume
              mountPath: /etc/coredns
            - name: kube-dns-config
              mountPath: /etc/kube-dns
      volumes:
        - name: xtables-lock
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
        - name: kube-dns-config
          configMap:
            name: kube-dns
            optional: true
        - name: config-volume
          configMap:
            name: node-local-dns
            items:
              - key: Corefile
                path: Corefile.base

exec into the pod:

kubectl exec -ti node-local-dns-cvljm -n kube-system sh

inside the pod:

iptables --list

iptables -L -vx
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment