Ahmad E aemneina

etcd performance testing and optimization

If your etcd logs start showing messages like the following, your storage might be too slow for etcd or the server might be doing too much for etcd to operate properly.

2019-08-11 23:27:04.344948 W | etcdserver: read-only range request "key:\"/registry/services/specs/default/kubernetes\" " with result "range_response_count:1 size:293" took too long (1.530802357s) to execute

If you storage is really slow you will even see it throwing alerts in your monitoring system. What can you do the verify the performance of your storage? If the storage is is not performing correctly, how can you fix it? After researching this I found an IBM article that went over this extensively. Their findings on how to test were very helpful. The biggest factor is your storage latency. If it is not well below 10ms in the 99th percentile, you will see warnings in the etcd logs. We can test this with a tool called fio which I will outline below.

Testing etcd per

working with etcd keys, notes

restore snapshot to work with

docker run --rm -v $(pwd):/data -e ETCDCTL_API=3 -w /data quay.io/coreos/etcd etcdctl snapshot restore snapshot.db

run snapshot to work with

docker run --name etcd -d -v $(pwd):/data -e ETCDCTL_API=3 -w /data quay.io/coreos/etcd

dump keys to debug

RKE2 commands

Updated on May 29 to accommodate etcd container not having /bin/sh available anymore.

Install

curl -sL https://get.rke2.io | sh
systemctl daemon-reload
systemctl start rke2-server

	#!/bin/bash
	export KUBECONFIG=$(pwd)/kube_config_cluster.yml
	IFS=$'\n'; for i in $(kubectl get secrets --all-namespaces \| grep "service-account-token"); do ns=$(echo $i \| awk '{print $1}'); sec=$(echo $i \| awk '{print $2}'); kubectl patch secret -n $ns $sec -p '{"metadata":{"finalizers": []}}' --type=merge && kubectl delete secret $sec -n $ns --wait=false; done
	IFS=$'\n'; for i in $(kubectl get pods --all-namespaces \| grep -v "NAME"); do ns=$(echo $i \| awk '{print $1}'); pod=$(echo $i \| awk '{print $2}'); kubectl delete pod $pod -n $ns --wait=false; done

	http_code: %{http_code}\n
	http_connect: %{http_connect}\n
	time_total: %{time_total}\n
	time_namelookup: %{time_namelookup}\n
	time_connect: %{time_connect}\n
	time_appconnect: %{time_appconnect}\n
	time_pretransfer: %{time_pretransfer}\n
	time_redirect: %{time_redirect}\n
	time_starttransfer: %{time_starttransfer}\n
	size_download: %{size_download}\n