Archana Shinde amshinde

Overview

The goal of this document to cover all aspects of Kubernetes management, including how resources are expressed, constrained and accounted for. This started a way to ensure that alternate container runtime implementation like Kata containers will behave from a resource accounting and consumption point of view in the same manner as runc.

Location of the latest version of this document: https://gist.github.com/mcastelino/b8ce9a70b00ee56036dadd70ded53e9f

If you do not understand cgroups please refer to a quick primer at the bottom of this document. This will help you understand how the resource enforcement actually works.

Kubernetes Resource Management

This is only a trace on vxlan interface on the host. not from within container namespace

  ping-26819 [001]  3678.931358: fib_table_lookup:     table 254 oif 0 iif 1 proto 17 0.0.0.0/40784 -> 10.244.2.4/1025 tos 0 scope 0 flags 0 ==> dev flannel.1 gw 10.244.2.0 src 10.244.0.0 err 0
  ping-26819 [001]  3678.931387: fib_table_lookup:     table 254 oif 0 iif 1 proto 17 10.244.0.0/40784 -> 10.244.2.4/1025 tos 0 scope 0 flags 0 ==> dev flannel.1 gw 10.244.2.0 src 10.244.0.0 err 0
  ping-26819 [001]  3678.931468: fib_table_lookup:     table 254 oif 0 iif 1 proto 1 0.0.0.0/0 -> 10.244.2.4/0 tos 0 scope 0 flags 0 ==> dev flannel.1 gw 10.244.2.0 src 10.244.0.0 err 0
  ping-26819 [001]  3678.931485: fib_table_lookup:     table 255 oif 0 iif 0 proto 0 0.0.0.0/0 -> 10.244.2.4/0 tos 0 scope 0 flags 0 ==> dev flannel.1 gw 10.244.2.0 src 10.244.0.0 err 0
  ping-26819 [001]  3678.931499: net_dev_queue:        dev=flannel.1 skbaddr=0xffff919236689800 len=98
  ping-26819 [001]  3678.931500: net_dev_start_xmit:   dev=flann

$ sudo bpftrace -e 'tracepoint:napi:napi_poll /comm == "ping"/ { printf("%s\n", comm) }'
Attaching 1 probe...
ping
ping
ping

$ sudo bpftrace -e 'tracepoint:napi:napi_poll /comm == "ping"/ { printf("%d\n", args->work) }'

Prepare the host

Get the image clear-XXXXX-kvm.img

wget https://download.clearlinux.org/releases/30080/clear/clear-30080-kvm.img.xz
unxz clear-30080-kvm.img.xz

Add intel_iommu=on to the kernel boot parameters

mkdir mount_dir

Despite changing time in the UI or through timedatectl, ubuntu seems to be reverting to the old timezone.

To change time and timezone in ubuntu:

sudo dpkg-reconfigure tzdata

Reference: https://askubuntu.com/questions/1092260/ubuntu-keeps-displaying-utc-time-despite-setting-timezone

Create Pod config json file with privileged set as true in the security_context. Without a priveged pod, containerd does not allow a privileged container to be run in the sandbox.

{
    "metadata": {
        "name": "priv-sandbox",
        "namespace": "default",
        "uid": "0"
    },
    "log_directory": "/tmp",

Install rust

https://rustup.rs

Install the rust language server and clippy tools:

$ rustup component add rustfmt rls
$ rustup component add clippy

	#!/bin/bash
	# -- mode: shell-script; indent-tabs-mode: nil; sh-basic-offset: 4; --
	# ex: ts=8 sw=4 sts=4 et filetype=sh

	VMN=${VMN:=1}

	NEMU=~/build-x86_64/x86_64-softmmu/qemu-system-x86_64

	sudo $NEMU \
	-trace events=/tmp/events \