docker disk space introspection kubernetes docker overlay

kubernetes_pods_docker_disk_usage.md

identify big pods/containers

investigage big files

DST=/mnt
find /var/lib -type f -size +1G -exec ls -lh {} \; | tee  $DST/bigfiles_var_lib_$(date "+%H%M").log
find /var/lib -type f -size +1G -exec ls -lh {} \; | awk '{ print $5 ": " $9 }' | sort -rh > $DST/bigfiles_var_lib_$(date "+%H%M").sorted.log

misbehave processes

lsof /dev/sda1 | grep deleted | tee $DST/lsof_deleted_with_open_descriptor_$(date "+%H%M").log

identify kubernetes pods with docker overlay2

With prometheus and cadvisor:

topk(100,container_fs_usage_bytes/(1000*1000*1000))

Record storage consumption

DST=/mnt
du -h --max-depth=2 /var/lib/docker/ | sort -k2 > $DST/var_lib_docker_$(date "+%H%M").log
du -h --max-depth=2 /var/lib/kubelet/ | sort -k2 > $DST/var_lib_kubelet_$(date "+%H%M").log

Overlay usage:

(not the best output, but works)

Note: du -x not to follow to different disk

TOP_STORAGE=$(du -hs /var/lib/docker/overlay2/* | grep -Ee '^[0-9]{3}[M]+|[0-9]G' | sort -h |tail -n 10 |tee -a /dev/stderr |awk '{print $2}'|xargs|sed 's/ /|/g')

docker inspect $(docker ps -q) | jq '.[]|.Config.Hostname,.Config.Labels."io.kubernetes.pod.name",.GraphDriver.Data.MergedDir,.hovno' | egrep -B2 "$TOP_STORAGE"

docker image inspect $(docker images -aq) | jq '.[]|.RepoDigests[0],.GraphDriver.Data.MergedDir,.hovno' |egrep -B2 "$TOP_STORAGE" 

# TOP_STORAGE=$(du -hs /var/lib/docker/overlay2/* | grep -Ee '^[0-9]{3}[M]+|[0-9]G' | sort -h |tail -n 10 |tee -a /dev/stderr |awk '{print $2}'|xargs|sed 's/ /|/g') 
522M    /var/lib/docker/overlay2/6853671b47d3ef99060f311925d3398e79e3bc26fe871d72e644b8069587c4b3
631M    /var/lib/docker/overlay2/3dcca422244861ff396899bc7d8e49f9b3d5f3bec33a58ddf2e8e2faf13e7e17
631M    /var/lib/docker/overlay2/45ca5c8ea8f752df6a554a735a25ab5009ffaf5724159a9e1a437d881d6bbb3f
631M    /var/lib/docker/overlay2/66fd27131f305496b6c603fb93f717560a50f3d756ee01ec1af035440cbabd7a
631M    /var/lib/docker/overlay2/72d3a0a17d55af360875867ef40e8e931b63d2cd5d05cf557acb2c9d29da813a
632M    /var/lib/docker/overlay2/368da01f2914f471122145bc2a47acbd388e7db6314ec9d5e721ae2297577f1a
634M    /var/lib/docker/overlay2/f04c9c1fae0db953e81ac7fa960a96e931683ae7474484d4a889ea69e53b84d1
658M    /var/lib/docker/overlay2/7537e271bc97c62241bc4186ebd1256aba2f80b8c5728c593b4205f9b6d7aa1f
684M    /var/lib/docker/overlay2/e7a120a7c2863d43518827aba0512a6ad0b3ece5569d1815df35fe10034da730
3.0G    /var/lib/docker/overlay2/8ca0d5c9c0ba097eb06f13bd345f788efc4c9fce59bf9d88acd973a5b86d06cc

# docker inspect $(docker ps -q) | jq '.[]|.Config.Hostname,.Config.Labels."io.kubernetes.pod.name",.GraphDriver.Data.MergedDir,.hovno' | egrep -B2 "$TOP_STORAGE"

"ver-pxcfv"
"/var/lib/docker/overlay2/f04c9c1fae0db953e81ac7fa960a96e931683ae7474484d4a889ea69e53b84d1/merged"
--
"master-2"
"ares2-848cc9459d-wfm5g"
"/var/lib/docker/overlay2/8ca0d5c9c0ba097eb06f13bd345f788efc4c9fce59bf9d88acd973a5b86d06cc/merged"
--
"master-2"
"obelix-nkhv2"
"/var/lib/docker/overlay2/7537e271bc97c62241bc4186ebd1256aba2f80b8c5728c593b4205f9b6d7aa1f/merged"
--
"master-2"
"kube-proxy-j45sh"
"/var/lib/docker/overlay2/72d3a0a17d55af360875867ef40e8e931b63d2cd5d05cf557acb2c9d29da813a/merged"
--
"master-2"
"kube-controller-manager-master-2"
"/var/lib/docker/overlay2/3dcca422244861ff396899bc7d8e49f9b3d5f3bec33a58ddf2e8e2faf13e7e17/merged"
--
"master-2"
"kube-scheduler-master-2"
"/var/lib/docker/overlay2/66fd27131f305496b6c603fb93f717560a50f3d756ee01ec1af035440cbabd7a/merged"
--
"master-2"
"kube-apiserver-master-2"
"/var/lib/docker/overlay2/45ca5c8ea8f752df6a554a735a25ab5009ffaf5724159a9e1a437d881d6bbb3f/merged"

Pod usage:

# just try
TOP_PODS=$(du -hs /var/lib/kubelet/pods/* | grep -Ee '^[0-9]{3}[M]+|[0-9]G' | sort -h |tail -n 10 |tee -a /dev/stderr|awk '{print $2}'|xargs|sed 's/ /|/g')

I get

Command 'jq' not found, but can be installed with:

sudo snap install jq  # version 1.5+dfsg-1, or
sudo apt  install jq

and then

E: Package 'jq' has no installation candidate

Maybe use some other utility?

I get

Command 'jq' not found, but can be installed with:

sudo snap install jq  # version 1.5+dfsg-1, or
sudo apt  install jq

and then

E: Package 'jq' has no installation candidate

Maybe use some other utility?

you can use https://github.com/itchyny/gojq