In the not-dockershim and not-CRIO normal socket path case, we are handled by the cri stats provider: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go
The 'magic' happens in the listPodStats function
Looping over each managed container, kubelet calculates the container statics at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go#L198, then calculate a running total of the pod usage at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go#L200, and then eventually return the results.
Initial potential issue I noticed was that we will run into is at the top of addPodCPUMemoryStats function. The
default action is to try to grab out the podCgroupInfo from cadvisor:
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go#L449-L456. If this is available, we simply
return. Of not, we go ahead and create running total for each of the containers' usages This could be okay, assuming
everything is sitting appropriately in the pod cgroup. I guess this is a chance where Kubelet can 'fast path' and not
need to do the pod usage += container. OK optimization, assuming podcgroup is accurate, though AFAICT it'll still go
through and grab these stats from CAdvisor "N" times, where N is the number of containers in the pod... not sure which is more expensive, a few CAdvisor calls or a couple of additions.
Second bigger issue occurs at the tail end of loop over all of the managed containers: "If cadvisor stats is available for the
container, use it to populate container stats". Umm; we just did a calculation in makeContainerStats and now, 8 lines later,
we overwrite the CPU and Memory values in the addCadvisorContainerStats call:
// If cadvisor stats is available for the container, use it to populate
// container stats
caStats, caFound := caInfos[containerID]
if !caFound {
klog.V(5).Infof("Unable to find cadvisor stats for %q", containerID)
} else {
p.addCadvisorContainerStats(cs, &caStats)
}
ps.Containers = append(ps.Containers, *cs)
This is definitely an issue, though it could be introduced by changes in Kata. We need to understand how a container is marked to say "cadvisor stats is available".
In our current scenario, the accurate value is overwritten with zero, resulting in kubectl top pod --containers=true always reporting zero.
We get stats from CRI here: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go#L151
We get stats directly from cgroups here: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go#L167
We do the right CRI based calculation here: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go#L198
And then we trash it with info from cgroups: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/stats/cri_stats_provider.go#L208