Maksim Paskal maksim-paskal

Class SRE Implements DevOps

Site Reliability Engineering (SRE) and DevOps are two ideas that have different origins but the same underlying objectives.

DevOps advocates for certain practices that increase success and productivity in a team building and running software. These include:

Reduce organization silos
Accept failure as normal
Implement gradual change
Leverage automation and tooling

Use dns cache in kubernetes

Install

https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/

to retrieve hostIP from pod add env to daemonset - dig kubernetes-node - will return hostIP of current pod

env:
- name: POD_HOST_IP
  valueFrom:

Simple envoy configuration with basic authentication and without authorization service

Sometime you need scrape prometheus metrics from external envoy that deploy not to kubernetes environment

You can use iptable or other stuff on external server to allow only trusted IP for scraping metrics - but for dynamic infrastructure some time it's hard to support it.

Envoy can expose this metrics more elegant style - using basic auth

Simple envoy.yaml

Simple bash script to change AMI in AWS EKS cluster

Sometime you need some not standart AMI in kubernetes cluster, for example you want to preload some docker images in kubernetes node

Fork https://github.com/awslabs/amazon-eks-ami
Make some changes in https://github.com/awslabs/amazon-eks-ami/blob/v20210208/scripts/install-worker.sh - for example

sudo systemctl start docker

	#!/bin/sh
	docker rm -f $(docker ps -qa)
	docker rmi -f $(docker images -q)
	docker volume rm $(docker volume ls -q)
	for mount in $(mount \| grep tmpfs \| grep '/var/lib/kubelet' \| awk '{ print $3 }') /var/lib/kubelet /var/lib/rancher; do umount $mount; done
	cleanupdirs="/etc/ceph /etc/cni /etc/kubernetes /opt/cni /opt/rke /run/secrets/kubernetes.io /run/calico /run/flannel /var/lib/calico /var/lib/etcd /var/lib/cni /var/lib/kubelet /var/lib/rancher /var/log/containers /var/log/pods /var/run/calico"
	for dir in $cleanupdirs; do
	echo "Removing $dir"
	rm -rf $dir
	done

	apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: maintance-pod
	spec:
	selector:
	matchLabels:
	app: maintance-pod
	replicas: 1
	template:

	PUT _cluster/settings
	{
	"persistent": {
	"cluster.routing.allocation.disk.threshold_enabled": true,
	"cluster.routing.allocation.disk.watermark.low": "93%",
	"cluster.routing.allocation.disk.watermark.high": "95%",
	"cluster.info.update.interval": "5m"
	}
	}

	http_filters:
	- name: envoy.lua
	typed_config:
	"@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
	inline_code: \|
	function stringSplit(inputstr, sep)
	if sep == nil then
	sep = "%s"
	end
	local t={}

	# TOP CPU Usage pods

	sort_desc(sum(rate(container_cpu_usage_seconds_total{image!="",container_name!="POD"}[5m])) by (pod,namespace,container_name,kubernetes_io_hostname))

	kubectl delete pods --field-selector=status.phase=Failed -A
	kubectl delete pods --field-selector=status.phase=Evicted -A
	kubectl delete pods --field-selector=status.phase=Succeeded -A