kubernetes_cert_prep.md

API Primitives

Object Spec:

• Provided to k8s
• Describes desired state of objects

Object status:

• provided by k8s
• describes the actual state of the object

More on APIs:

• All yaml will be converted to JSON by the k8s
• all yaml require an API version: apiVersion: v1
• All yamls require
• • spec
• • version
• • kind
• • metadata
• Common k8s objects: nodes, pods. deployments, services, configmaps

Names:

• Unique names are client provided, can be reused, legnth of 253 chars

Namespaces:

• multiple virtual clusters backed by the same virtual cluster
• provide scope for names.
• easy way to divide cluster resources with resource quotas.

Nodes:

• services necessary to run pods
• managed by master components
• necesary services:
• • Container runtime (docker, crio)
• • Kubelet
• • Kube-proxy
• not created by k8s, stood up by cloud providers or yourself

Cloud controller Mangers:

• route controller
• service controller
• persistentvolumelabels controller

NodeController:

• assigns CIDR block to newly registered node
• keeps track of nodes
• monitors node health
• evicts pods from unhealthy nodes
• can taint nodes based on current conditions in more recent versions.
• 40 second node timeout, 5 minutes to redirect pod creation.
• "Node Eviction Behavior" - checks to see percentage of nodes that need to be rescheduled.
• "Node Eviction Rate" - implemented per availability zone.
• can create node TAINTS to label a node as not ready

Kubernetes services

• pod is the simplest k8s object, represents one or more containers running on a SINGLE node
• Ephemeral, disposable & replaceable - they are stateless
• "Cattle vs pets"
• Usually managed by deployments

Deployment Specs:

• image
• number of replicas
• services -> deployments
• particular port or IP address

Services continued:

• running the application pods
• how you set up a service depends on networking conf and how you wil handle load balanacing and port forwarding.
• kube-proxy redirects traffic

imperitive:

• kubectl run nginx --image=nginx
• declarative: (putting the complete yaml file)
• apiVersion helps future proof.

no namespace given? it assumes 'default'

• Services expose deployments.
• Pods are managed by deployments

Application & Persistent Storage

• apt install nfs-kernel-server -y ; mkdir -p /var/nfs/general; chown -R nobody:nogroup /var/nfs/general; echo "/var/nfs/general * (rw,sync,no_subtree_check)"" >> /etc/exports
• on each node: apt install nfs-common

Example YAML to setup the PersistentVolume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: lapv
spec:
  capacity:
    storage: 1Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Recycle # wiped when a user relinquishes the claim on it.
  nfs:
    path: /var/nfs/general
    server: 10.0.1.128
    readOnly: false

PersistentVolumeClaim Example

apiVersion:v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

Example PVC pod

apiVersion: v1
kind: Pod
metadata:
 name: nfs-pod
 labels:
   name: nfs-pod
spec:
 containers:
 - name: nfs-ctn
   image: busybox
   command:
     - sleep 
     - "3600"
   volumeMounts:
   - name: nfsvol
     mountPath: /tmp
 restartPolicy: Always
 securityContext:
   fsGroup: 65534
   runAsUser: 65534
 volumes:
   - name: nfsvol
     persistentVolumeClaim:
       claimName: nfs-pvc

Authentication & Authorization

• Authorization:
• • ABAC
• • RBAC
• • WebHook
• Does not have a native User object in the object store.

Kublet Auth

• How to authorize access
• • By default, requests are treated as anonymous requests
• • if authenticated, then it authorized the request
• • Default is AlwaysAllow
• • Might want to subdivide access because
• • • Anonymous auth enabled but anonymous users should be limited
• • • bearer token auth enabled, but some service accounts should be limited
• • • client certificate auth enabled but only some that are signed should be allowed

Cluster DNS

• Allows us to use hostnames to connect services. Runs as kube-dns on the master node as a docker container.
• kubectl exec -it sleep -- nslookup kubernetes
• must be exposed in order for other pods to discover services.

Get info about a node

• kubectl describe node k8snode1
• kubectl get pods --all-namespaces -o wide
• kubectl get pods -n kube-system # just the overhead k8s processes
• kubectl get pods -o wide # get detailed reports on nodes (ip addresses etc)

Kubernetes Network Policies

• Specifications on how groups of pods may communicate.
• Implemented by the network plugin
• pods are non-isolated by default

sudo kubeadm init --pod-network-cidr=10.244.0.0/16
kubeadm join --token --discovery-token-ca-cert-hash sha256:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config 

sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

kubectl get pods --all-namespaceskubectl get pods --all-namespaces
# Daemon Sets
* Used when a process should be run on every node
* * `kubectl get daemonsets -n kube-system` # list the daemonsets for a specified namespace
* Annotations: not used to identify like labels, more like notes.

# Defining security contexts
```yaml
spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 200
...
  securityContext:
    allowPrivilegeEscalation: false

Delete pods

• kubectl delete -f nginx.yaml # file method
• kubectl delete pod nginx-pod delete by pod name

Deployments

• kubectl get deployments
• kubectl create -f nginx-deployment.yaml #Notice that when adding a deployment, you use create instead of apply
• kubectl describe deployment kube-dns
• kubectl get deployment kube-dns -o yaml # return a yaml file describing the deployment

Rolling Updates

• kubectl set image deployment nginx-deployment nginx=nginx:1.8. # this allows you to modify a deployment rather than add the entire yaml again
• kubectl rollout status deployments/nginx-deployment # listst the status of the rollout event
• • kubectl rollout history deployments/nginx-deployment --revision 3 # view the history

Rollbacks

• kubectl rollout undo deployments/nginx-deployment --to-revision=2
• kubectl rollout undo deployment nginx-deployment # if you want to undo the last deployment

find where a pod is running

kubectl get pods -l app=nginx -o wide

End to end testing

• Primarily a developer tool

• built with ginko and gomega.

• Kubetest suite can be used like minimesos

• Pod networking application such as falnnel is needed to allow the pods to communicate

• • Makes use of an overlay network (vxlans by default) to provide that service.

High Availability

• Monit- Restart the kubelet using systemd

• Storage Layer - Make sure you have rock solid persistent storage.

• Be sure to run a replicated etcd cluster.

• replicated api services

• • mkdir /srv/kubernetes on each node. add keys and certs
• • if you alraedy have a working master, just copy the /srv/k8s/*yaml over. The kubelet process will notice and automagically kickstart the apiServer

• REMEMEBER. This will require any services to talk to the load-balancer rather than just the hardcoded IP of the original master.

• Now we need to allow our state to change.

• • Controller managers and scheduler.

• *Theseprocesses must not modify the clusters state at the same time, use a lease-lock

• • Each scheduyler and ciontroller manager can be launched with a --leader-elect flag.
• • The scheulder and controller-manager can be configured to talk to the API server that is on the same node LOCALHSOT
• • --leader-elect-flag - the scheduler and cronteoller-manager will comlete the leader election process mentioned befopre with the flag is used.
• • Installing configuration files

• create empty log files so docker will mount the files and not make new ones: touch /var/log/{kube-scheduler,kube-controller-manager}.log

• Copy the kube-{manager,scheduler}.yaml files into the /etc/k8s/manifests directory

• And now yoru cluster is highly available!

How kubernetes configures applications

Config Maps

• kubectl create conigmap my-map --from-literal=school=LinuxAcademy
• kubectl get configmaps
• kubectl describe configmap my-map # describe a configmap
• kubectl get pods --show-all # show completed pods
• `kubectl logs config-test-prod # return logs from a ran container

Ingress

• API object that manages external access to the services in a cluster. Usually HTTP.
• Can include load balancing and SSL. Think of it as a gateway.
• Edge-router.
• Collection of rules to allow external hosts to consume internal services
• `kubectl get ing
• Securing ingress
• • TLS/Cert
• • Port 443
• • Multiple hosts are miltplexed on the same port by hostnames specified through the SNI TLS extenstion.
• • TLS secret must contain keys named tls.crt and tls.key that contain the certificate and privcate key to use for TLS
• If you didnt want to use an ingress controller, you could:
• • Use Service.Type=LoadBalancer
• • Use Service.Type=NodePort
• • User a port proxy. `

Labels

• kubectl get pods -l app=nginx # view things by label only.
• kubectl label pod mysql-asdfadsf test=sure --overwrite
• • kubectl describe pod -l test=sure
• • kubectl label pods -l app=nginx tier=frontend # label mutiple pods at one time.
• • kubectl label pods --all -n default tier=linuxAcademyCloud # Label all the pods of a specified namespace a certain label.

Deploying a load-balancer

• Happens async
• "kind:service", selector is a label targeting mechanism
• "Type: LoadBalancer"

Monitoring

Managing Logs

• kubectl logs <podname> # this only does STDOUT logs
• stackdriver | fluentd allow you to send logs to a centralized logging server.
• kubectl exec -it counter -- whoami # run a command in a running pod

Node Networking Configuration

Master Nodes

Port Range	Service
6443	K8s API Server
2379-2380	Etcd server client API
10250	Kubelet API
10251	Kube-scheduler
10252	kube-controller-manager
10255	Read-only kubelet API

Worker Nodes

Port Rance	Service
10250	Kubelet API
10255	Read-only kubelet API
30000-32767	NodePort Service
10.0.1.135 k8snode2
10.0.1.138 k8snode1
10.0.1.136 k8smaster

Persistent Volumes

• Native pod storage is ephemeral - like a pod
• When a container crashes:
• • kubelet restarts it
• • file system is re-created from image
• • ephemeral files are gone

Docker volumes

• directory on disk
• possibly in another container
• new volume drivers

Kubernetes Volumes

• same lifetime as a pod
• data preserved across container restarts
• pod goes away -> volume goes away
• directory with data
• volumes cannot mount onto other volumes
• no hard links to other volumes
• each pod must specify where each volume is mounted

Empty dir

• Created when a pod is assigned to a node
• exists while pod runs on a particular node
• initally empty
• multiple containers can read/write same volume
• pod removed -> volume removed
• optional: emptyDir.medium=memory creates a ram backed filesystem

GlusterFS

• Multiple concurrent mounts -- read/write -- are allowed.

hostPath

• mounts file or directory from host nodes filesystem to a pod

Mount Propagation

• allows for the sharing volumes mounted by one container to other containers in the same pod
• other pods in the same node
• --feature-gates MountPropagation=true
• mountPropagation subfield:
• • hosttocontainer - container gets subsequent mounts to this volume (default)
• • bidirectional - hostToContainer, plus host sees subsequent mounts made by container

Volumes and their access modes

• persistentvolume - api for users that abstracts implementation details of storage
• persistantvolumeclaim - mehtod for users to claim durable storage

PersistentVolume (PV)

• cluster resource, provisioned storage in the cluster
• volume plugins have independent lifecycle from pods
• volumes share the lifecuycle fo the pod

PersistantVolumeClaim

• request for storage
• pods consume node resources; PVCs consume PV resources;
• Pods can request specific CPU and memory; PVCs can request specific size and access modes.
• PV & PVCs have a set lifecycle
• • provision
• • bind
• • reclaim
• Provisioning
• Static
• • Creates PVs
• • in the k8s api and available for consumption
• Dynamic
• • used when none of the static PVs match the PVC
• • based on storageclasses
• • pvc must request a created and configured storage class
• • claims requesting nameless class disable dynamic provisioning. *Binding
• • user creates pvs
• • master watches for new pvcs and matches them to pvs
• • binds are exclusive
• • claims not matched will remain unbound indefinitely.

Resource limits and pod scheduling

• FOLLOW UP: What is a taint? - It allows for a node to repel work.
• • Taints allow a node to repel a set of pods.
• Toleration: allows you to override a taint
• `kubectl taint nodes node-role.kuberetes.io/master-

Resources

• if your pod exceeds the amount of memory allocated, it may consume more than allocated but it is subject to termination if need be. `

Scaling applications

• kubectl get deployments; kubectl describe deployments neginx-deployment
• kubectl scale deployment/nginx-deployment --replicas=5
• Note: there is a 5 minute timeout after a node dies.

Securing Images

• Apply security updates
• Dont run tools like apt-update in containers
• use rolling updates
• ensure only autherized images are used in your environment
• use private registries to store approved images
• CI pipeline should ensure that only vetted code is used for building images

Security

• K8s has several well thoughout, pre created roles
• RBAC- role based access control.
• Unlike role-based access control (RBAC), which employs pre-defined roles that carry a specific set of privileges associated with them and to which subjects are assigned, the key difference with ABAC is the concept of policies that express a complex Boolean rule set that can evaluate many different attributes.
• kubelets expose https endpoints. default is wiiiiide open. we can and should secure them with kubelet auth. x509 cert req
• • --anonymous-auth=false
• Network Policies (per namespace) - restrict network access. networking cni must support these polices.
• users can be assigned quotas or limit ranges.
• gaining access to etcd is very very bad.
• enable audit logging. (beta) archive that audit file off to a secure server
• You can set up automated credential lifetimes.

Service Networking - Exposing pods to the outside world

• kubectl expose deployment <deployment_name> --type="NodePort" --port 80
• kubectl get service
• Utilizes kubeproxy to proxy requests from the worker nodes to the master.

TLS for cluster components

• easyrsa init-pki
• easyrsa --batch "--req-cn=${MASTER_IP}" build-ca nopass # Create a certificate authority
• Certificates go into /etc/kubernetes/pki/{ca.crt}