cri stats windows

stats.md

cri-only stats

KEP: kubernetes/enhancements#2371 KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2371-cri-pod-container-stats

containerd linux implementation: containerd/containerd#6113

crictl implementation: kubernetes-sigs/cri-tools#886

cri pr: https://github.com/kubernetes/kubernetes/pull/102789/files

kubelet implementation: https://github.com/kubernetes/kubernetes/pull/103095/files

how to test

use containerd 1.6+ crictl 1.23+

using ctr:

.\ctr.exe -n k8s.io t metrics 514b78f4758d2eb00d03c43c00af84c1567024f67b4abc6d898f31aa10f278dd

using crictl (doesn't work with windows yet):

./crictl statsp

containerd metrics calls stats endpoint

cri stats calls task.Metrics:

https://github.com/containerd/containerd/blob/e3db7de8f53769a4b7cef81154f5eebe21430219/pkg/cri/server/container_stats_list.go#L39

metrics interface implemented by local stats implementation:

https://github.com/containerd/containerd/blob/6b35307594841829c398ee8cf1100538867fcb82/services/tasks/local.go#L617

calls task stats:

https://github.com/containerd/containerd/blob/6b35307594841829c398ee8cf1100538867fcb82/services/tasks/local.go#L671

hcs shim implements stats

https://github.com/microsoft/hcsshim/blob/9efc65486d7ab7599b751a5035e718c4330ec8f0/cmd/containerd-shim-runhcs-v1/service.go#L419

which ends up calling

https://github.com/microsoft/hcsshim/blob/a750a7fa43a5c93e736d3e1cc12d8b6b04ca60fe/cmd/containerd-shim-runhcs-v1/task_hcs.go#L930\

kubelet implementation

usagenanocores is calculated from usagecorenanoseconds and saved to cache https://github.com/kubernetes/kubernetes/blob/de37b9d293613aac194cf522561d19ee1829e87b/pkg/kubelet/stats/cri_stats_provider.go#L741

https://github.com/kubernetes/kubernetes/blob/5cb6fab8f6336990cc58f7cb6abffc6ad178121b/staging/src/k8s.io/kubelet/pkg/apis/stats/v1alpha1/types.go#L204-L207

// Total CPU usage (sum of all cores) averaged over the sample window. // The "core" unit can be interpreted as CPU core-nanoseconds per second. // +optional UsageNanoCores *uint64 json:"usageNanoCores,omitempty"

TODO

need to add stats relevant to windows: https://github.com/kubernetes/kubernetes/pull/102789/files#diff-583fcb2b2b9fed64e8ac45392565a924918b4a64a7d4b66df0d4bccd298e8769R630

pod stats as of today don't really add up for Windows:

`k get --raw "/api/v1/nodes/capz-conf-5nptc/proxy/stats/summary" | jq .pods[].cpu`  

sometimes pod usage nanocores is 0, sometimes nanocores is not
There are no networkstats
missing a bunch of memory stats

https://github.com/kubernetes/kubernetes/blob/de37b9d293613aac194cf522561d19ee1829e87b/pkg/kubelet/stats/cri_stats_provider.go#L207

container accepts pods with no uid, but crictl doesn't allow deleting with out uid

references

https://github.com/kubernetes/kubernetes/pull/105744/files