system pre-requisites sriov setup

K8S-SYSTEM-SETUP-UBUNTU-1604.md

Initial Setup On Fresh Ubuntu 16.04 Install

Run the below commands as root user

sudo su

Common setup

Disable swap

sed -i '/swap/s/^/#/g' /etc/fstab

Install Docker

bash -c "apt-get update && apt-get -y install docker.io"

systemctl enable docker && systemctl start docker

Install k8s pre-req

sudo apt-get update && sudo apt-get install -y apt-transport-https

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF

sudo apt-get update && sudo apt-get install -y kubelet=1.10.* kubeadm=1.10.* kubectl

Enable CPU manager in k8s, needed for DP

cat << "EOF" > /etc/systemd/system/kubelet.service.d/0-cpu-manager.conf
[Service]
Environment="KUBELET_EXTRA_ARGS=--cpu-manager-policy=static --cpu-manager-reconcile-period=5s --kube-reserved=cpu=500m"
EOF

Convenience setup

Configure mgmt interface to come up on boot with dhcp

cat > /etc/systemd/network/dhcp.network << "EOF"
[Match]
Name=eno*

[Network]
DHCP=yes
EOF

systemctl enable systemd-networkd

Configure perms of stack user, home and groups (customize for your user)

MYUSER=stack
chown -R $MYUSER:$MYUSER /home/$MYUSER
usermod -aG docker $MYUSER

sed -i -e '/Defaults\s\+env_reset/a Defaults\texempt_group=sudo' /etc/sudoers
echo "$MYUSER ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/99_$MYUSER
chmod 440 /etc/sudoers.d/99_$MYUSER

Performance setup

Configure hugepages (customize, here 32G) and IOMMU. Make sure BIOS has VT-d enabled

sed -i '/GRUB_CMDLINE_LINUX_DEFAULT/c\GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=32"' /etc/default/grub
update-grub

Mount hugepages on boot

echo 'nodev /dev/hugepages hugetlbfs pagesize=1GB 0 0' | tee -a /etc/fstab

Install lastest i40e driver (and firmware, if necessary. Not described here)

apt-get install -y make gcc

I40E_VER=2.4.10
wget https://downloadmirror.intel.com/24411/eng/i40e-${I40E_VER}.tar.gz && \
tar xvzf i40e-${I40E_VER}.tar.gz && cd i40e-${I40E_VER}/src && make install && cd -

Setup vfio-pci module to load on boot, dpdk and sriov script

echo 'vfio-pci' | tee /etc/modules-load.d/vfio-pci.conf
wget -qO- https://fast.dpdk.org/rel/dpdk-17.11.2.tar.xz | tar -xJC /opt
mv /opt/dpdk-* /opt/dpdk

mkdir -p /sriov-cni /opt/scripts
cat << "EOF" > /opt/scripts/sriov.sh
#!/bin/bash
# Copied from infra/sriov.sh
# Usage: ./sriov.sh ens785f0

NUM_VFS=$(cat /sys/class/net/$1/device/sriov_totalvfs)
echo 0 | tee /sys/class/net/$1/device/sriov_numvfs
echo $NUM_VFS | tee /sys/class/net/$1/device/sriov_numvfs
sudo ip link set $1 up
for ((i = 0 ; i < ${NUM_VFS} ; i++ )); do ip link set $1 vf $i spoofchk off; done
for ((i = 0 ; i < ${NUM_VFS} ; i++ )); do ip link set dev $1 vf $i state enable; done
EOF

# Script perms
sudo chmod 744 /opt/scripts/sriov.sh

Setup SRIOV on ens785f0 and ens785f1 for sriov-cni to use

# Systemd unit to run the above script
cat << "EOF" > /etc/systemd/system/sriov.service
[Unit]
Description=Create VFs for ens785f0 ens785f1

[Service]
Type=oneshot
ExecStart=/opt/scripts/sriov.sh ens785f0
ExecStart=/opt/scripts/sriov.sh ens785f1

[Install]
WantedBy=default.target
EOF

# Enable the SRIOV systemd unit
systemctl enable sriov

Reboot

reboot