set master node
user@miguelmota1:~$ sudo kubeadm init --pod-network-cidr=10.244.0.0/16
[sudo] password for user:
^B^[kk[init] using Kubernetes version: v1.11.2
[preflight] running pre-flight checks
I0901 00:44:36.182370 16584 kernel_validator.go:81] Validating kernel version
I0901 00:44:36.182463 16584 kernel_validator.go:96] Validating kernel config
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [miguelmota1.mylabserver.com kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.31.113.178]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [miguelmota1.mylabserver.com localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [miguelmota1.mylabserver.com localhost] and IPs [172.31.113.178 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 39.001439 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node miguelmota1.mylabserver.com as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node miguelmota1.mylabserver.com as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "miguelmota1.mylabserver.com" as an annotation
[bootstraptoken] using token: vvp5zn.6eu7gkbno8yngxzf
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 172.31.113.178:6443 --token vvp5zn.6eu7gkbno8yngxzf --discovery-token-ca-cert-hash sha256:236a2cad7496e888be3268124b071d8496b01b7927373a1f6ad8c28a96e4087d
user@miguelmota1:~$ kmkdir -p $HOME/.kube^C
user@miguelmota1:~$ mkdir -p $HOME/.kube
user@miguelmota1:~$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
user@miguelmota1:~$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
^ will copy config files to a local directory
pods are small unit of compute
user@miguelmota1:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-78fcdf6894-6gvml 0/1 Pending 0 1m
kube-system coredns-78fcdf6894-mgbn8 0/1 Pending 0 1m
kube-system etcd-miguelmota1.mylabserver.com 1/1 Running 0 57s
kube-system kube-apiserver-miguelmota1.mylabserver.com 1/1 Running 0 1m
kube-system kube-controller-manager-miguelmota1.mylabserver.com 1/1 Running 0 1m
kube-system kube-proxy-qv4kd 1/1 Running 0 1m
kube-system kube-scheduler-miguelmota1.mylabserver.com 1/1 Running 0 1m
scheduler determines which nodes will host which containers as they come in
join node to a cluter (on server 2)
sudo kubeadm join 172.31.113.178:6443 --token vvp5zn.6eu7gkbno8yngxzf --discovery-token-ca-cert-hash sha256:236a2cad7496e888be3268124b071d8496b01b7927373a1f6ad8c28a96e4087d
show nodes in cluster (on server 1)
user@miguelmota1:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
miguelmota1.mylabserver.com NotReady master 6m v1.11.2
miguelmota2.mylabserver.com NotReady <none> 40s v1.11.2
user@miguelmota1:~$
kube-proxy runs on each node to provide network services
kube master has kube-apiserver kube-apiserver uses etcd key value store for settings kube-scheduler is responsible for pods cloud-controller-manager is responsible for persistent storage and routing
nodes has a kubelet (takes orders from master) node has a kube proxy node has a pod a pod runs a container
kubernetes objects are persistent entities in kubernetes system kubernetes objects are "records of intent" describes
- what applications are running
- which nodes those applications are running on
- policies around those applications object spec
- provided to kubernetes
- describe desired state of objects object status
- provided by kuberentes
- describe the actual state of the object kubctrl turns the yml file to an api request common kubernetes objects
- nodes
- pods
- deployments
- services
- configmaps namespaces are virtual clusters namespaces allow for
- resource quotes
- multiple teams of users node
- any worker machine (prev called minions)
- can run pods
- managed by master
- kubelet orchestrates the containers cloud controller managers
- route controller (gce clusters only)
- service controller
- PersistentVolumeLabels controller node controller
- assigning CIDR block to newly registered node
- keeps track of nodes
- monitors the node health
- evicts pods from unhealthy nodes (graceful termination) pod
- simplest kubernetes object - represents one or more containers running on a single node
- stateless and disposable services refer to deployments (port or ip) kubernetes service is imperative (do this thing) - ex: kubectl run nginx --image=nginx can also be declarative: using yml file no namespace uses "default" namespace as the namespace pods are managed by deployments services expose deployments third parties handle loadbalancing and port forwarding to those services, through ingress objects
kubectl describe job pi
kubectl logs pi-fmctx
create a pod
kubectl create -f alpine.yaml
delete pod
kubectl delete -f alpine.yaml
delete pod (other way)
kubectl delete pod alpine
delete pod (other way)
kubectl delete pod/alpine
connection between apiserver and nodes, pods and services are unencrypted therefore unsafe to run over public networks nodes are not inherently created by kubernetes, nodes are added to a cluster and kubernetes object is created to reflect them the master controls the kubernetes cluster kubernetes primites are pod,service,persistentVolume,and deployment. communication between apiserver and kubelet on the clusetr are not used for keep-alive xml packets data formats for kubernetes api call are JSON and YML containers run on nodes in typical deployment the kubernetes master listens on port 443 a pod represents a running process difference between docker volume and kubertnetes volume is that in docker it is loosely defined while in kubernets the volume hash the same lifetime as its surrounding pod MemoryPressure is a key that will be true if memory is running low unique ip addresses are assigned to pods a kubelet mounts volumes to containers Minikube is the recommended method for creating a single node kubernetes deployment on your local workstation Minkube doesn't require a cloud provider kubeadm can be used to deploy a multi-node locally but it's a little more challenging. need to select Cluster Network Interface (CNI) if going this route
get status of node and list DiskPressure and MemoryPressure statuses
kubectl describe node <node-name>
list all pods and which nodes they are currently running on
kubectl get pods --all-namespaces -o wide
list pods running in the kube-system namespace
kubectl get pods -n kube-system
Flannel is a pod networking application to allow pods to communicate vxlans is the technology flannel uses
Cluster communications
- cover communication
- everything in kubernetes goes through the api driver
- TLS is default encryption
- most installations handle the certificate creation
- kubadm created certs
- anything that connect to the API, including nodes, proxies, the scheduler, volume plugins should be authenticated
kubernetes has role-base access control (RBAC)
- certain roles perform specific actions in the cluster
kubelets expose https endpoints which give access to both data and actions on the nodes. By default they are open
- to secure endpoints, enable kubelet authentication by starting with "--anonymous-auth=false" and assigning x509 client cert
kubernetes uses etcd for configuration and secrets, it acts as the k/v store for entire cluster gaining write access to etcd is equivalent of gaining root on the whole cluster isolate etcd behind a firewall and only allow requests from the API servers rotate credentials frequently don't allow third parties into the kub-system namespace
setting HA
- create reliable nodes that will form cluster
- set up redundant and reliable storage service with a multinode deployment of etcd
- start replicated and load balanced kubernetes api servers
- set up a master-elected kubernetes scheduler and controller manager daemons everything the talks to the api must go through the load balancer step one
- make the master node reliable
- ensure services automatically restart if they fail
- kubelet already does this
- if kubelet goes down , need something to restart it
- "monit" on debian systems or systemctl on systemd-based systems step 2
- etcd already replicates storage to all master nodes
- to lose data, all three nodes would need disk failures
- increase 3 to 5 for more reliability can use clustered file system like Gluster or Ceph, or RAID array on each physical machine step three
- create the initial log file touch /var/log/kube-apiserver.log
- create a /srv/kubernetes/ directory on each node which should include:
- basic_auth.csv - basic auth user/pwd
- ca.crt
- known_tokens.csv - tokens that entities (ie the kubelet) can use to talk to the apiserver
- kubecfg.crt - client cert, pub key
- kubecfg.key - client cert, priv key
- server.crt - server cert, pub key
- server.key - server cert, priv key
- copy kube-apiserver.yaml into /etc/kubernetes/manifests on each of the master nodes
- the kubelet monitors the directory and automatically creates an instance of the kube-apiserver using the pod definition specified in the file step four
- allow state to change
- controller managers and scheduler
- these processes must not modify the clusters state simultaneously, use a lease-lock
- each scheduler and controller manager can be launched with a "--leader-elect" flag
- scheduler and controller-manager can be configured to talk to the api server that is on the same node or to a load balanced IP of api server
create empty log files on each node so that docker will mount the files and not make new directories
- touch /var/log/kube-scheduler.log
- touch /var/log/kube-controller-manager.log set up descriptions of scheduler and controller manager pods on each node by copying the kube-scheduler.yaml and kube-controller-manager.yaml into the /etc/kubernetes/manifests/
- if worker goes down, kubernetes will detect and spin up replacement pods
kubetest is a testing suite for kubernetes e2e testing Ceph is an object store Canal, WeaveNet are all CNI providers CNI must enforce the network policies The master runs the apiserver must choose a CNI when deploying kubernetes with kubeadm
kubectl describe deployment nginx-deployment
output deployment yaml
kubectl describe deployment nginx-deployment -o yaml
update the deployment image
kubectl set image deployment/nginx-deployment nginx=nginx:1.8
see update status
kubectl rollout status deployment/nginx-deployment
update the deployment image using a yml file
kubectl apply -f nginx-deployment.yaml
view current deployments
kubectl get deployments
view deployment history revisions
kubctl rollout history deployment/nginx-deployment --revision=2
revert to a revision
kubectl rollout undo deployment/nginx-deployment --to-revision=2
get pod by label
kubectl get -l app=nginx -o wide
get status of pods
kubectl get pods name-of-pods -o wide
create a k/v map
kubectl create configmap my-map --from-literal=school=LinuxAcademy
get list of maps
kubectl get configmaps
list k/v of map
kubectl describe configmaps my-map
output map as yml
kubectl get configmap my-map -o yaml
use logs sub cmd to display environment variables
kubectl create -f pod-config.yaml
kubectl get pods --show-all
kubectl logs config-test-pod
use configmap to decouple configuration from yaml file
scale up number of replicas (pods)
kubectl scale deployment/nginx-deployment --replicas=3
environment variables are used to configure an application in a container
kubectl delete pod podname will spin up another pod to match number of replicas
Always, OnFailure, Never are valid restart policies
cpu:"250m" is how you limit cpu utilization to one quarter (m is for millicpus)
DaemonSet is used from a CNI container that needs to run on every node
pod labels are used to assign a pod to a particular node
pods make up deployments. services point to deployments.
labels are used to select and identify objects. annotations allow for a wider variety of characters that labels do not allow. both use k/v pair config maps
set a label on a pod
kubectl label pod mysql-foobar test=sure --overwrite
get information by label
kubectl describe pod-l test=sure
taints label nodes that are going to repel work
untaint
kubectl taint nodes my-node node-role.kubernetes.io=master-
taint
kubectl taint nodes my-node node-role.kubernetes.io=master:NoSchedule
deploy pods to particular node (ie has hardware requirements)
first tag node (ie net tag)
kubectl label node node1 net=gigabit
"nodeSelector" is a pod property which you can set the label to which deploy to
show info for only pod running
kube describe pod
the "schedulerName" tag in the spec can be used to specify which scheduler a pod should use. defaults to "default-scheduler"
a scheduler is a pod on the master node
the pod will not be scheduled until a node with the resources becomes available if the pod request more resources than available
taints are used to repel certain pods from nodes and are applied to nodes
two pods to have anti-affinity can be used to run on different nodes to avoid sharing failure domains
podAffinity is used for placing two or more pods on the same node
the scheduler determines which node will be used to instantiate the new pod
annotations are important when using multiple schedulers bc they remind operators which scheduler was used to place or fail to place a pod. annotations are used to provide additional non-identifying info about a pod and things like app version or scheduler that placed the pod
if toleration and a taint match during scheduling then the taint is ignored and the pod might be scheduled to the node Tolerations are applied to pods and allow the pod to schedule onto nodes with matching taints One or more taints applied to a node marks the node should not accept any pods that do not tolerate the taint.
"heapster" provides cluster-wide aggregator of monitoring and event data. runs as a pod
cAdvisor is an open source container resource usage and performance analysis agent. runs on port 4194
get logs of container in pod
kubectl logs pod-name
log location in systemd based os
/var/log/containers
tail logs
kubectl logs podname -f
kubectl logs podname --tail=10
kubectl logs podname --tail
/var/log/pods is where the kubernetes k/v store (etcd) logs live
kubectl exec mypod --cat /var/log/applog
get shell prompt to container
kubectl exec -it mypod --container sidecar1 -- /bin/bash
view metrics
kubectl top [nodes | pods]
get logs back from a dead pod
kubectl logs --previous
updating
sudo apt update kubelet
upgrade kubadm
kubeadm upgrade apply v1.9.1
drain pods
kubectl drain nodename --ignore-daemonsets
systemctl status kubelet
kubectl uncordon mynode
must evict pod to run on different node when trying to update node
list tokens
sudo kubeadm token list
sudo kubeadm token generate
sudo kubeadm token create <token from prev cmd> --ttl 3h --print-join-command
"uncordon" allows the scheduler to once again allow pods to be scheduled on the node
any drains that cause the number of ready replicas to fall below the specified budget are blocked
"Node Self Registration Mode" is the mode cluster should be to add more nodes. When the kubelet flag "--register-node=true" (default) is set the kubelet will attempt to register itself with the API server.
append "--runtime-config=api/all=false,api/v1=true" to use only v1 api
run kubeadm again when nodes fail when upgrading, it is idempotent
execute command inside of container
kubectl exec mypod /usr/bin/id
"Termination Messages" are logs about fatal events
Authorization Methods are ABAC, RBAC, Webhook
Admission control modules can access the content of objects
default authorization mode is "AlwaysAllow"
network policy is how pods communicate with each other
use labels to select pods and define rules
by default pods accept connections from everyone
network policies are implemented by the network plugin
"podSelector" property in yaml file "metadata" name: allow-all "metadata" name: default-deny
container security context takes precedence over pod security context
a Pod Security Policy is a cluster-level resource that controls security sensitive aspects of the pod specification
The PodSecurityPolicy objects define set of conditions that a pod must run in order to be accepted by the system
pod,container are the levels security context can be applied to
admins can limit a user in a namespae by creating a new role in the users namespace with appropriate rules
expose port
kubectl expose deployment webhead --type="NodePort" --port 80
"NodePort" means all node ports in the entire cluster
show exposed ports
kubectl get services
kube-proxy redirects to the appropriate node in the cluster
ingress is an api object that manages external access to the services in a cluster. ingress could be a gateway managed by a cloud provider
a service is a kubernetes service that identifies a set of pods using label selectors
ingress are services and pods that have IPs only routable by the cluster network ingress is a collection of rules that allow inbound connections
users request ingress by POSTing to the apiserver
most cloud providers deploy an ingress controller on the master. each ingress pod must be annotated with the appropriate class so that kubernetes knows that's an ingress controller.
metadata:
annotations:
ingress.kubernetes.io/rewrite-target:/
show ingress rules
kubectl get ing
secure an ingress with a secret which contains tls private key (tls.key) and cert (tls.crt)
data:
tls.crt: <base64>
tls.key: <base64>
kind: Secret
ingress controller is bootstrap with a load balancing policy that applies to all ingress objects
health checks are not exposed directly through the ingress
edit ingress with default editor
kubectl edit ing <name>
deploy a load balancer
type: LoadBalancer
get pods by kubernetes
kubectl get pods -n kube-system
get pods ran by user
kubectl get pods
dns entries are created automatically when the service is created
show deployments
kubectl get deployments
make dns resolvable
kubectl expose deployment dns-target
set a label on a node
kubectl label node mynode foo=bar
shorthand for services
kubectl get svc
deploy a service
kubectl expose deployment deployment-name
start a deployment
kubectl create -f deployment.yml
cm is shorthand for configmap
schedule a pod only on a node with the label "network=gigabit'
apiVersion:v1
kind:Pod
metadata:
name:
spec:
containers:
-name:
image:
nodeSelector:
network:gigabit
recycling policies of PVs
- Retain (keep the contents)
- Recycle (scrub the contents)
a kubelet is the control plane that runs on the nodes
get all recent events sorted by their timestamp
kubectl get events --sort-by=".metadata.creationTimestamp"
delete all objects created by a file
kubectl delete -f mistake.yml
po is abbriviation for pod
return a node to the service
kubectl uncordon mynode
create an nginx deployment with replicas without using yml
kubectl run nginx --image=nginx --replicas=3
ds is abbreviation for DaemonSet
all kubernetes yaml begin with these tags
apiVersion:
Kind:
metadata:
kubernetes object has a "spec" which acts as the "record of intent". The other main part is the "status"
shutdown a deployment
kubctl delete deployments/my-deployment
edit a live pod
kubectl edit pod mypod
set default editor with
KUBE_EDITOR="nano"
deploy is the abbreviation for deployment
get back the yaml describing a deployment
kubectl get deployment mydeployment -o yaml
delete everything under a namespace include the namespace
kubectl delete namespace mynamespace
service is a set of running pods that work together
the scheduler determines where to deploy pod
replication controller is a loop the drives the current state towards desired state
liset secrets
kube get secretes
"restartPolicy: Never" will only run task once. Used for db migrations
see running jobs
kubectl get jobs
show list of ips of all the pods (routed through load balancer)
kubectl describe service myservice
containers within a pod can read eac others ports
flanneld allocates subnet leases to each host flanneld runs on each host via DaemonSet
a cloud provider that supports kubernetes-provision load balancers is required to specify a service type of "LoadBalancer"
an ingress controller compatible with available and appropriate service providers like load balancers is required to request an ingress resource
ingress in an api object
network policies determines how set of pods are allowed to communicate with each other
the CNI handles inter-pod communication
ingress was introduced in kubernetes 1.1+
all traffic is sent to a single host if an ingress request is made with no associated rules
the result of service type of ClusterIP is a single IP address within the cluster that redirects traffic to a pod (possibly on a different node) serving the application (the pod)
ClusterIP is most commonly used with 3rd party load balancers
order is Preamble,podSelector,ingress,egress
.local is how pods can resolve the hostname
native pod storage is ephemeral
spec.volumes indicates which volumes to provide for the pod
spec.ontainers.volumeMounts indicates where to mount these volumes in the containers
volumes cannot mount onto other volumes
CSI = Container Storage Interface
"downwardAPI" mounts a directory and writes data in plain text files
"emptyDir" - created when a pod is assigned to a node. exists only when pod runs on a particular node
a container crashing does not delete storage from a pod
persistentVolume - api for users that abstracts implementation details of storage
persistentVolumeClaim - method for users to claim durable storage regardless of implementation
Persistent Volume (PV)
- provisioned storage in the cluster
- do not share lifecycle of pod
PersistentVolumeClaim (PVC)
- request for storage
- pod consumes node resources
- pod can request specific cpu and memory requirements
PVs and PVCs have a set lifecycle
- provision
- bind
- reclaim
storage AccessModes
- ReadWriteOnce - can be mounted as r/w by one node only (RWO)
- ReadOnlyMany - can be mounted read-only by many-nodes (ROX)
- ReadWriteMany - can be mounted r/w by many nodes (RWX)
make directory to share with pods
sudo mkdir -p /var/nfs/general
chown nobody:nogroup /var/nfs/general
sudo vim /etc/exports
# add:
# /var/nfs/general <local ip of node>(rw,sync,no_subtree_check)
# then restart
sudo systemctl restart nfs-kernel-serverinstall nfs on each node (including master)
sudo apt install nfs-common