Kubernetes Memory Metrics

kubernetes_memory_metrics.md

Kubernetes Memory Metrics Explained

MemTotal: all memory installed or provisioned to the system
MemFree: memory without anything in it (wasted)
SReclaimable: This is the part of the slab or “in-kernel data structures cache” that can be reclaimed.
ActiveFile: Amount of buffer or page cache memory that is in active use.
InactiveFile: Amount of buffer or page cache memory that are free and available.
WatermarkLow: Placeholder memory

MemAvailable: A more accurate estimate of how much memory is available. (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=34e431b0a)
    = MemFree + ActiveFile + InactiveFile + SReclaimable
      - MIN((ActiveFile + InactiveFile) / 2, WatermarkLow)
      - MIN(SReclaimable / 2, WatermarkLow)
      - WatermarkLow

MemUnavailable: Memory that is not available for use (kubectl top node).
    = MemTotal - MemAvailable

MemFuzzyUsed: A fuzz value for efficient access, see https://github.com/google/cadvisor/issues/3286#issue-1648530485

MemWorkingSet: Active memory usage by the container (kubectl top pods / container_memory_working_set_bytes), also USED to decide if node has enough memory.
    = MemFuzzyUsed - InactiveFile

MemoryCapacity: Total amount of memory available on the node.
SystemReserved:  Memory reserved for OS system daemons like sshd, udev, etc., disabled by default.
KubeReserved: Memory reserved for Kubernetes system daemons like the kubelet, container runtime, etc., disabled by default.
EvictionThreshold: Memory threshold, when MemWorkingSet > MemoryAllocatable first pod will be evicted on a node, https://github.com/kubernetes/kubernetes/issues/43916#issue-218553580

MemoryAllocatable: Total amount of memory that CAN BE allocated to Kubernetes Pods.
    = MemoryCapacity - System-Reserved - Kube-Reserved - Eviction-Thresholds

kubectl top node usage = MemUnavailable per node (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
kubectl top pod usage = MemWorkingSet per pod (container_memory_working_set_bytes = memory.usage_in_bytes - total_inactive_file)
https://www.redhat.com/en/blog/using-oc-adm-top-to-monitor-memory-usage

Example:
    Machine highmem-2 => 16GiB RAM

    MemoryCapacity = 15933.941Mi
    SystemReserved = 1024Mi
    KubeReserved = 0
    EvictionThreshold = 200Mi

    MemoryAllocatable = 15933.941Mi - 1024Mi - 0 - 200Mi = 14709.941Mi

bash function to get node and pod memory values

function get_nodes_memory_capacity(){ echo "$(cluster=$1; echo "NODE TOTAL_MEMORY_CAPACITY TOTAL_ALLOCATABLE_MEMORY UNAVAILABLE_MEMORY ALLOCATABLE_AVAILABLE_MEMORY WORKING_SET_MEMORY ALLOCATABLE_WORKING_SET_MEMORY"; echo "$(kubectl --context=$cluster get nodes --no-headers | grep -E "Ready" | awk '{print $1}' | while IFS=' ' read -r node; do echo "$node $(echo "$(kubectl --context=$cluster get node $node -o jsonpath='{.status.capacity.memory}')" | sed -E 's/([0-9]+)([a-zA-Z]+)/\1 \2/' | awk '{gi=$1/1024; print gi "Mi"}';) $(echo "$(kubectl --context=$cluster get node $node -o jsonpath='{.status.allocatable.memory}')" | sed -E 's/([0-9]+)([a-zA-Z]+)/\1 \2/' | awk '{gi=$1/1024; print gi "Mi"}') $(echo "$(kubectl --context=$cluster top node $node --no-headers)" | awk '{print $4}') $(echo "$(echo "$(kubectl --context=$cluster get pod -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,STATUS:.status.phase,NODE:.spec.nodeName --all-namespaces)" | grep -E "Running" | grep $node | awk '{print $1}' | paste -sd '|' - | while IFS= read -r pods; do kubectl --context=$cluster top pod --all-namespaces | grep -E "$pods" | awk '{print $4}' | sed 's/Mi$//' | awk '{pods_mem_sum += $1} END {print pods_mem_sum}'; done;)" | awk '{print $1 "Mi"}')"; done;)" | awk '{gsub(/Mi/, "", $3); gsub(/Mi/, "", $4); available=$3 - $4; allocwork=$3 - $5;print $1 " " $2 " " $3 "Mi " $4 "Mi " available "Mi " $5 " " allocwork "Mi"}';)" | column -t;}; echo ""; get_nodes_memory_capacity <context>

NODE                                 TOTAL_MEMORY_CAPACITY  TOTAL_ALLOCATABLE_MEMORY  UNAVAILABLE_MEMORY  AVAILABLE_MEMORY  WORKING_SET_MEMORY  ALLOCATABLE_WORKING_SET
........8ff4e-pool01-c1de302d-fmxxt  15933.9Mi              14709.9Mi                 6593Mi              8116.9Mi          2924Mi              11785.9Mi
........8ff42-pool01-c1de302d-v92jz  15933.9Mi              14709.9Mi                 8687Mi              6022.9Mi          4320Mi              10389.9Mi