Krzysztof Wilczyński kwilczynski

Kubernetes DNS at scale

It seems that is a common practice in HPC and AI/ML environments that use MPI applications to populate a hosts files with all the nodes on the cluster and copy it over all the nodes, ref https://help.ubuntu.com/community/MpichCluster

It is my observation that in Kubernetes, Headless Services are used to implement this Service Discovery This is very handy because it allows to reference a pod by hostname without having to copy over a generace /etc/hosts.

There must also be an A record of the following form for each ready endpoint with hostname of and IPv4 address . If there are multiple IPv4 addresses for a given hostname, then there must be one such A record returned for each IP.

By default Linux distros are unoptimized in terms of I/O latency. So, here are some tips to improve that.

Most apps still don't do multi-threaded I/O access, so it's a thread-per-app which makes per-app speed always bottlenecked by single-core CPU performance (that's not even accounting for stuttering on contention between multiple processes), so even with NVMe capable of 3-6 GB/s of linear read you may get only 1-2 GB/s with ideal settings and 50-150/100-400 MB/s of un/buffered random read (what apps actually use in real life) is the best you can hope for.

All writes are heavily buffered on 3 layers (OS' RAM cache, device's RAM cache, device's SLC-like on-NAND cache), so it's difficult to get real or stable numbers but writes are largelly irrelevant for system's responsiveness, so they may be sacrificed for better random reads.

The performance can be checked by:

`fio --name=read --readonly --rw={read/randread} --ioengine=libaio --iodepth={jobs_per_each_worker's_command} --bs={4k/2M} --direct={0/1} --num

BPF Cheatsheet

Download binary

curl -s -L https://github.com/libbpf/bpftool/releases/download/v7.2.0/bpftool-v7.2.0-amd64.tar.gz | tar xvzf - -C /tmp chmod +x /tmp/bpftool

List network

Random numbers in bash et al

There is a variable $RANDOM than you can read in bash or zsh to get a random number beteen 0 and 32767. Here in zsh on my (old) mac:

% echo $RANDOM
13757
% echo $RANDOM
16896

	* PCI legacy (from https://lore.kernel.org/all/[email protected]):

	Fixes:
	Closes: (link or Message-ID]
	Suggested-by:
	Link:
	Reported-by:
	Tested-by:
	Co-developed-by: (co-author)
	Signed-off-by: (co-author)

	cd ./ccache/build
	make clean
	export PATH='/usr/local/bin:/usr/local/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
	ccmake -DDEPS=DOWNLOAD -DCMAKE_BUILD_TYPE=Release -DENABLE_TESTING=OFF -DREDIS_STORAGE_BACKEND=OFF -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_SYSCONFDIR=/etc ..
	make -j$(nproc)
	make DESTDIR='/tmp/ccache' PREFIX='/usr' install

	ccflags-y := -DDEBUG -Wfatal-errors

	ifneq ($(KERNELRELEASE),)
	obj-m += state-toggle.o
	else
	BUILD_KERNEL ?= /lib/modules/$(shell uname -r)/build
	default:
	$(MAKE) -C $(BUILD_KERNEL) M=$(CURDIR) modules
	endif

	# See for more options https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
	apiVersion: apps/v1
	kind: DaemonSet
	metadata:
	name: node-ethtool
	namespace: kube-system
	labels:
	k8s-app: node-ethtool-config
	spec:
	selector:

	Fixes:
	Closes:
	Suggested-by:
	Link:
	Reported-by:
	Tested-by:
	Co-developed-by: (co-author)
	Signed-off-by: (co-author)
	Signed-off-by: (author)
	Signed-off-by: (chain)

	# -- mode: ruby --
	# vi: set ft=ruby :

	# Vagrant box for testing
	Vagrant.configure("2") do \|config\|
	config.vm.box = "fedora/35-cloud-base"
	memory = 6144
	cpus = 4

	config.vm.provider :virtualbox do \|v\|