Kubernetes notes for CKA exam preparation 💪

kubedingdong.md

This notes applied to Kubernetes version 1.9.1-00 on Ubuntu Xenial.

Basic concept

TO BE WRITTEN

Installation and starting up 🏁

Installation

• Swap should be disabled (see /etc/fstab)
• Docker, apt-get install -y docker.io
• Add Kubernetes repository, deb http://apt.kubernetes.io/ kubernetes-xenial main to /etc/apt/sources.list.d/kubernetes.list
• Add the gpg key of the repo, curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
• Install Kubernetes tools (kubeadm, kubelet), apt-get update && apt-get install -y kubeadm=1.9.1-00 kubelet=1.9.1-00
• To get the autocomplete feature on bash, source <(kubectl completion bash)

Starting (master node)

• Initialize, kubeadm init --apiserver-advertise-address=172.20.10.8 --pod-network-cidr 10.244.0.0/16. 172.20.10.8 is the specific public IP that will be used as apiserver endpoint. 10.244.0.0 is the internal network that used by pods to communicates.
• Follow the instruction to run kubernetes as regular user, or just mkdir -p ~/.kube && sudo cp /etc/kubernetes/admin.conf ~/.kube/config && sudo chown $(id -u):$(id -g) ~/.kube/config
• At this point you are in regular user and the node should be NotReady (see kubectl get node)
• Prepare the network configuration file, for instance use Flannel
• wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
• Set the Network on the configuration, example : set it to 10.244.0.0 as intialized on kubeadm init before.
• kubectl apply -f kube-flannel.yaml
• At this point, regular user should be able to use kubectl command seamessly, but kubeadm still need to be executed from root user.
• Check frequently with kubectl get node
• Check the node's detail, kubectl describe node nodename
• Check pods, kubectl get pods --all-namespaces
• Tada!

Starting (other node)

Preparation on the master node

• Get token value from sudo kubeadm token list
• Get the Discovery Token CA cert hash from openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex

On the regular node

• kubeadm join --token a37497.be623ec61a8122dd 10.0.2.15:6443 --discovery-token-ca-cert-hash sha256:6581dc2a13c1848943b4297c0ffc121cfb6b894a083321673aeac6e9a374ee48
• In case the master node refused to connect (getsockopt : connection refused error), look at the result of kubectl get svc

Simple deployment and scaling 👶

• kubectl run omama --image nginx
• kubectl get deployments
• kubectl get deployment omama -o yaml > omama.yaml
• Edit the file and add these line below under container's name line :

...
ports:
- containerPort: 80
...

• Delete the existing deployment, kubectl delete deployment omama
• Apply the new deploy conf, kubectl apply -f omama.yaml
• Scaling, kubectl scale deployment omama --replicas=3, the scale count represented on pods' count.
• Deleting a pod should recreate the new pod, up to the replication conf, kubectl delete pod omama-xxxxx, the new pod will be created with initial status ContainerCreating
• Expose the port, kubectl expose deployment/omama
• Look at the services and endpoint, kubectl get svc,ep
• Log into the child node, try to curl the nginx webserver default page, curl 10.244.1.12:80
• Deleting the deployment, remove deployment, svc, and ep by the deployment name, example : kubectl delete deployment omama

Explosing the service outside the cluster

• See the pods, kubectl get pods
• Remove the related service, kubectl delete svc omama
• Recreate the service with LoadBalancer type, kubectl expose deployment omama --type=LoadBalancer
• See the port on services, kubectl get svc omama

NAME      TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
omama     LoadBalancer   10.106.16.55   <pending>     80:32576/TCP   2m

• Access the cluster service with the specific port through the browser : http://10.106.16.5:32576 🚀

Stopping

• For non maste node : kubectl drain --force nodename
• Unschedule the node : kubectl cordon nodename
• Stop the service (or turn of the machine) : systemctl stop kubelet

Reset (master)

• Remove virtual interface, ip link delete virtualinterfacename

kubeadm reset
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
sudo rm -rf /etc/kubernetes/manifests/*
sudo rm -rf /var/lib/etcd/*

Miscellaneous

Setup a virtual network interface based on existing physical interface

root@x1carbon:/home/herpiko/src/lfs258# ifconfig wlp4s0:0
wlp4s0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 60:57:18:0c:77:db  txqueuelen 1000  (Ethernet)

root@x1carbon:/home/herpiko/src/lfs258# ifconfig wlp4s0:0 192.168.1.1
root@x1carbon:/home/herpiko/src/lfs258# ping 192.168.1.101

Another way to get as shell access of pod's container ❓

• You may face this issue, kubernetes-retired/kubernetes-anywhere#337
• Inspect the pod with describe
• You will see the Node and Container ID values.
• Enter the node and use docker to access the container.

Operation

Resource limits

• kubectl run hog --image vish/stress
• kubectl get deployment hog -o yaml > hog.yaml
• Add the resource limit, like this :

resources:
  limits:
    memory: 3Gi
  request:
    memory: 2500Mi
  terminationmessagePath: /dev/termination-log
  terminationMessagePolicy: File

• Replace the current deployment, kubecl replace -f hog.yaml
• Confirm the replacement, kubectl get deployment hog -o yaml
• Modify the yaml again (don't forget to remove status and other unique ID, we'll apply this instead of replace)

        resources:
          limits:
            cpu : 1
            memory: 4Gi
          requests:
            cpu: 0.5
            memory: 2500Mi
        args:
        - -cpus
        - "2"
        - -mem-total
        - "950Mi"
        - -mem-alloc-size
        - "100Mi"
        - -mem-alloc-sleep
        - "1s"

• kubectl delete deployment hog
• kubectl apply -f hog.yaml
• The cpu usage on node 1 should be increased.
• Modify the resource again, set the CPU (limits, requests) to more than the resource that you have, re-apply.
• See pods, the deployment should be failing.
• See the log with kubectl logs hog-xxxxx to see what's going wrong.

Resource limits on namespace

• Create new namespace, kubectl create namespace low-usage-limit
• Check the namespaces available, kubectl get namespaces
• Create a LimitRange configuration on an yaml file, low-resource-range.yaml

apiVersion: v1
kind: LimitRange
metadata:
  name: low-resource-range
spec:
  limits:
  - default:
      cpu: 1
      memory: 500Mi
    defaultRequest: 
      cpu: 0.5
      memory: 100Mi
    type: Container

• Create the LimitRange, kubectl create -f low-resource-range.yaml --namespace=low-usage-limit
• Check the available LimitRange, kubectl get LimitRange --all-namespaces. Not specifying any namespace will return no resources.
• Run the hog again on the low-usage-limit namespace, kubectl run limited-hog --image vish/stress -n low-usage-limit
• Check the result, kubectl get deploy,pods -n low-usage-limit
• Delete the deployment
• Redeploy a normal hog, export the yaml, add the namespace low-usage-limit, then apply. The deployment should inherite the resource limit from the namespace / LimitRange conf.

API

TLS access

• Isolate the ca, cert and key from ~/.kube/config to txt file, then decode it, cat ca.txt | base64 -d > ca.pem. Now you should've ca.pem, cert.pem and key.pem on your filesystem.
• See the apiserver address on these config file too.
• Try to access the apiserver, curl --cert cert.pem --key key.pem --cacert ca.pem https://192.168.1.100:6443/ap/v1/pods. You should get the expected result.
• You can explore more about the API here, ~/.kube/cache/discovery/192.168.1.100_6443
• These API consumed by kubectl command through openat, see strace kubectl get node

RESTful API access

• kubectl config view to find the server entry, will get c5c82d8aca5e1a312d077506f50a720a000eb7cbe1b0f862c8b3339887664767IP address and port.
• kubectl get secrets --all-namespaces to get the bearer-token.
• Lets take a look to default-token-xxx, kubectl describe secret default-token-xxx
• Copy the token to access the cluster API, curl https://192.168.1.101/apis --header "Authorization: Bearer inserttokenstringhere" -k
• The cert also available inside the pod's container. ❓

Or using a proxy

• kubectl proxy --api-prefix=/
• Then try to curl without any auth, curl localhost:8001/api/v1/namespaces

Cronjob ⏰

• Create the yaml file, cron-job.yaml

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: date
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: dateperminute
            image: busybox
            args:
            - /bin/sh
            - -c
            - date; sleep 30
          restartPolicy: OnFailure

• Create it, kubectl create -f cron-job.yaml
• Check, kubectl get cronjob
• Watch,kubectl get jobs --watch
• The jobs will done in it's own pods, kubectl get pods -a
• Delete the cronjob if you wish, kubectl delete cronjob date

ReplicaSets 🗿 🗿 🗿

• Create the ReplicaSet configuration, named rs.yaml

apiVersion: extions/v1beta1
kind: ReplicaSet
metadata:
  name: rs-one
spec:
  replicas: 2
  template:
    metadata:
      labels:
        system: ReplicaOne
      spec:
        containers:
        - name: nginx
          image: nginx:1.7.9
          ports:
          - containerPort: 80

• Fire! kubectl create -f rs.yaml
• Inspect, kubectl get rs, kubectl decribe rs rs-one, kubectl get pods
• Delete ReplicaSet instance without aspecting pods, kubectl delete rs/rs-one --cascade=false
• If we create the ReplicaSet again, the new ReplicaSet will take the ownership of the current pods, kubectl create -f rs.yaml
• We can isolate a pod from it's ReplicaSet, kubectl edit pod rs-one-xxxxx
• Change the system value to IsolatedPod then save the file.
• The pod will be excluded from the ReplicaOne and a new pod will be created to fulfill the ReplicaOne's replication count.
• Lets delete the ReplicaOne, kubectl delete rs rs-one then check the pods again.
• All the pods will be flushed except the IsolatedPod's pod
• It's also possble to delete pod by system label, kubectl delete po -l system=IsolatedPod

DaemonSet

• DaomenSet will ensure the the pods will be deployed to each node. ❓ Need more understanding
• Copy the rs.yaml to ds.yaml, change Replica string to Daemon and remove the replicas:2 line.
• Create the DaemonSet, kubectl create -f ds.yaml
• Try to join a new node to the cluster, see if the new pod get deployed on these particular node.

Rolling Updates and Rollbacks

• Inspect the previous DaemontSet, specific to update strategy, kubectl get ds ds-one -o yaml | grep -A 1 Strategy, the result will be like this :

  updateStrategy:
    type: OnDelete

• Which means the container will be upgraded when the predecessor is deleted. Lets try it.
• Update the nginx image on ds-one, kubectl set image ds ds-one nginx=nginx:1.8.1-alpine
• Check the current pod, kubectl describe po/ds-one-xxxx | grep image:, should be nginx:1.7.9
• Delete the current pod and inspect the new one, the nginx version should be upgraded to 1.8.1-alpine. Pooh!
• Any changes on DaemonSet will be recorded, see it with kubectl rollout history ds/ds-one
• Inspect the specific revision, kubectl rollout history ds/ds-one --revision=1
• To rollback to revision 1, kubectl rollout undo ds/ds-one --to-revision=1, just delete the current pod. The new pod will be deployed using the nginx 1.7.9 back.
• There is also another strategy named RollingUpdate. Lets try it.
• Copy the current ds, kubectl get ds ds-one -o yaml > ds-two.yaml herpiko@ubuntu-node-master:~$ kubectl get node --show-labels NAME STATUS ROLES AGE VERSION LABELS
• Change OnDelete string to RollingUpdate on ds-two.yaml, then deploy it.
• Inspect the ds-two pods, it's on nginx 1.7.9.
• Edit the ds-two directly, kubectl edit ds/ds-two, change the image to nginx:1.8.1-alpine, save it.
• Inspect the pods again, it should on nginx 1.8.1-alpine.
• Deleting ds is just like deleting another type of objects, kubectl delete ds ds-one,ds-two, the pods will be deleted too.

NodeSelector and labels (8.1)

• Cerate a nginx-one.yaml, please pay attention to the nodeSelector value.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-one
  labels:
    system: secondary
  namespace: accounting
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:1.7.9
        imagePullPolicy: Always
        name: nginx
        ports:
        - containerPort: 8080
          protocol: TCP  
      nodeSelector:
        system: secondOne

• If you tried to create the object, it'll fail since there is no namespace named accounting. Create the namespace first.
• If you check the pods, no pod are got deployed (it stay Pending) since there are no nodes that labeled with secondOne
• Check the labels of the nodes, kubectl get nodes --show-labels
• Lets add a label to one of the node, kubectl label node ubuntu-node-1 system=secondOne
• Check the labels of the nodes again, then check the pods. They should got deployed properly.
• If you want to remove label from a node, use minus sign, kubectl label onde ubuntu-node-1 system-
• It's also possible to assign pods to master node using labels. For example, some deployment need more resources and master node has it. You also can check the deployed container within docker to make sure the pods are deployed on the correct node.

NodePort

• Delete the previous deployment (nginx-one), then redeploy again.
• Expose with NodePort type service, kubectl expose deployment nginx-one --type=NodePort --name=service-lab -n accounting
• Inspect the service, kubectl describe svc -n accounting, see the NodePort value.
• Try to access the public IP of the node with the NodePort's value as port. ❓ The exercise lab said that it could be accessible from master node's IP too

ConfigMap

• Try to create some files :

$ mkdir primary
$ echo c > primary/cyan
$ echo m > primary/magenta
$ echo y > primary/yellow
$ echo k > primary/black
$ echo "known as key" >> primary/black
$ echo blue > favorite

• These files (and it's content) will be mapped to a ConfigMap, lets create it.
• kubectl create configmap colors --from-literal=text=black --from-file=./favorite --from-file=./primary/
• See the configmap resources, kubectl get configmaps
• See the mapped values, kubectl get configmap colors -o yaml
• Lets create a pod that use the values from colors

apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
spec:
  containers:
  - name: nginx
    image: nginx
    env:
    - name: ilike
      valueFrom:
        configMapKeyRef:
          name: colors
          key: favourite

• After deployed, you can see that the blue value can be fetched inside the container. ❓ 1.9.1 has issue with kubectl exec

Persistent NFS Volume (PV)

On master node

• Install nfs server, sudo apt-get install -y nfs-kernel-server
• Create the dir to be shared, sudo mkdir /opt/sfw && sudo chmod 1777 /opt/sfw && sudo echo "software" > /opt/sfw/hello.txt
• Add this line to /etc/exports

/opt/sfw/ *(rw,sync_no_root_squash,subtree_check)

• Reread the conf, sudo exportfs -ra

On child node

• Install the nfs client, sudo apt-get install -y nfs-common
• Take a look to the mountpoints of master node, showmount -e 192.168.1.100
• Try to mount, sudo mount 192.168.1.100:/opt/sfw /mnt
• Check it, ls -l /mnt/ && cat /mnt/hello.txt

Back to the master node to create pv object

• Write this file pvol.yaml then fire it.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvvol-1
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /opt/sfw
    server: 192.168.1.100
    readOnly: false

• Check the pv resources, kubectl get pv

Persistent NFS Volume Claim (PVC)

• Create the object file, pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-one
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 200Mi

• Create it, kubectl create -f pvc.yaml
• The pvc shoud be bound to pv pvvol-1, kubectl get pvc,pv
• Lets create deployment that use this pv, nfs-pod.yaml,

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: 2018-03-04T17:14:36Z
  generation: 1
  labels:
    run: nginx
  name: omama
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        volumeMounts:
        - name: nfs-vol
          mountPath: /opt
        ports:
        - containerPort: 80
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      volumes:
      - name: nfs-vol
        persistentVolumeClaim:
          claimName: pvc-one
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30

• After got deployed, you can take a sneak peek to the container to make sure the NFS is mounted to /opt

ResourceQuota to Limit PVC Count and Usage

• Remove any previous deployment, pv and pvc.
• Create ResourceQUota object, named storage-quota.yaml

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storagequota
spec:
  hard:
    persistentvolumeclaims: "10"
    requests.storage: "500Mi"

• Create a new namespace named small
• Deploy the pv, pvc and storagequota (sequentially) on these namespace
• Inspect the namespace, you'll see that there are Resource Quotas.
• Remove namespace line from nfs-pod.yaml to allow us to assign another namespace, deploy it on small namespace.
• After the pod got deployed, check again the Resource Quotas on namespace's detail.
• Create any empty file that has 300mb size on /opt/sfw then check again the Resource Quoatas on small namespace.
• Lets simulate what happens when a deployment requests more than the quota. Remove any deployment and check the ns's detail. The storage did not get cleaned when pod was shutdown.
• Delete the pvc, pv STATUS should be released.
• Delete pv, change PersistentVolumeReclaimPolicy to Delete, the create again.
• The current quota on ns should be zero.
• Create pvc again.
• Remove the current resourcequota, change 500Mi to 100Mi, create it again.
• Verify the current quota on ns, the hard limit has already been exceeded.
• Create deployment, no error seen, check if the pods actually running.
• Remove the deployment then check the pv. Status should be Failed since there's no deleter volume plugin for NFS.
• Delete any pv, pvc and deployments, the change pv's persistentVolumeReclaimPolicy value to Recycle
• Apply the low-resource-range.yaml to the small namespace
• Create the pv again (now on Recycled mode)
• If we tried to create the pvc, it should return an error because of exceeded quota. Change the requests.storage to 500Mi with kubectl edit resourcequota -n small then create pvc again. It should work this time.
• Create deployment again, then inspect ns.
• Delete pvc, the STATUS of pv should be Released then Available
• ❓ Need more understanding of Retain, Delete, and Recycle mode difference

Taints ❓ Need a beast machine

• You can control the pods spreads on nodes with taints. There are 3 modes : NoSchedule, PreferNoSchedule, NoExecute. They will be applied to the next deployment but only NoExecute will force to move the pods to another nodes in instance.
• Set taint to a node, kubectl taint nodes ubuntu-node-2 bubba=value:NoSchedule. I dunno why the key is bubba or is it required to be bubba
• Check the taint on the specific node, kubectl describe node ubuntu-node-2 | grep Taint
• To remove taint from node, kubectl taint nodes ubuntu-node-2 bubba-
• You can try redeloy again and again (using simple deployment with replica count more than 8) to see the effect of each taint mode on each node.

Logs

• Logs are in :
  • journalctl -u kubelet
  • sudo find /var/log -name "*apiserver*log"
  • kubectl logs podID
  • kubectl get events

Custom Resource Definition

Helm and Charts

TLS

Auth

Adminission controllers

❌ : not covered

• secret ❌
• taint
• namespace
• reschedule pod to another node
• replica
• daemonset
• configmap
• dnsrecord ❌
• investigating/log/issues
• rolling update rollback
• etcdctl snapshot via cert
• rbac ❌

https://github.com/feiskyer/kubernetes-handbook/blob/master/en/troubleshooting/pod.md