Skip to content

Instantly share code, notes, and snippets.

@Kambaa

Kambaa/kubeadm-cluster-installation-steps.md

Last active January 7, 2025 10:25
Show Gist options
  • Save Kambaa/24753f49b12bbb98f7c342f683292fcf to your computer and use it in GitHub Desktop.
Save Kambaa/24753f49b12bbb98f7c342f683292fcf to your computer and use it in GitHub Desktop.
Download ZIP
Kubernetes Installation Steps on Ubuntu Server 24.04 (tested with virtualbox osboxes image)
Raw
kubeadm-cluster-installation-steps.md

Kubernetes Cluster Installation Steps

1. osboxes ubuntu-server virtualbox image settings:

sudo dpkg-reconfigure keyboard-configuration

70 for generic keyboard 93 for Turkish layout (press 1 for the other defaults)

2. Prerequisites:

  • Update the system

    sudo apt update -y && sudo apt upgrade -y
sudo apt-get dist-upgrade -y

  • install necessary tools

    • nano: text editor
    • apt-transport-https: allows the use of repos accessed via https
    • gnupg: secures data and communication
    • ca-certificates: manages cert authorities
    • curl: data transfer from/to a server
    • software-properties-common: abstraction for managing apt repos
    • inetutils-traceroute: traceroute
    • ethtool: controlling network drivers and hardware
    • iptables: ip packet filtering
    • openssh: ssh server
    sudo apt install -y nano apt-transport-https gnupg ca-certificates curl software-properties-common inetutils-traceroute ethtool iptables

sudo apt install openssh-server -y
sudo systemctl enable --now ssh

  • Disable swap (docs)

    # disable swap temporarily
swapoff -a 

# disable swap permanently
sudo nano /etc/fstab
# comment out swap line with #

  • Enable IPv4 packet forwarding (docs)

    # sysctl params required by setup, 
# params persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
EOF

# Apply sysctl params without reboot
sudo sysctl --system

# Verify that net.ipv4.ip_forward is set to 1 with:
sysctl net.ipv4.ip_forward

  • Add overlay and br_netfilter modprobes:

    # enable temporarily
sudo modprobe overlay
sudo modprobe br_netfilter

    add/open the file /etc/modules-load.d/k8s.conf

    # enable permanently
sudo nano /etc/modules-load.d/k8s.conf

    add these lines

    overlay
br_netfilter

    and reboot

3. necessary installation files (download links for offline installation):

File Github URL
containerd.io_1.7.24-1_amd64.deb https://download.docker.com/linux/debian/dists/bookworm/pool/stable/amd64/containerd.io_1.7.24-1_amd64.deb
crictl-v1.31.0-linux-amd64.tar.gz https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.31.0/crictl-v1.31.0-linux-amd64.tar.gz
kubeadm https://cdn.dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubeadm
kubelet https://cdn.dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubelet
kubelet.service https://raw.githubusercontent.com/kubernetes/release/v0.16.2/cmd/krel/templates/latest/kubelet/kubelet.service
10-kubeadm.conf https://raw.githubusercontent.com/kubernetes/release/v0.16.2/cmd/krel/templates/latest/kubeadm/10-kubeadm.conf
kubectl https://dl.k8s.io/release/v1.32.0/bin/linux/amd64/kubectl

4. install containerd and crictl

  • download and install containerd

    curl -o containerd.io_1.7.24-1_amd64.deb https://download.docker.com/linux/debian/dists/bookworm/pool/stable/amd64/containerd.io_1.7.24-1_amd64.deb

sudo dpkg -i containerd.io_1.7.24-1_amd64.deb

  • Configure containerd to use the systemd for cgroups. : docs

    sudo mkdir -p /etc/containerd

sudo touch /etc/containerd/config.toml

sudo containerd config default > /etc/containerd/config.toml

sudo nano /etc/containerd/config.toml

    find SystemdCgroup and set it to true ( containerd official docs, k8s docs ) :

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

    sudo systemctl restart containerd

  • install crictl:

    Optionally install crictl (required for interaction with the Container Runtime Interface (CRI), optional for kubeadm)crictl-v1.31.0-linux-amd64.tar.gz, github download link:

    DOWNLOAD_DIR="/usr/local/bin"
sudo mkdir -p "$DOWNLOAD_DIR"
CRICTL_VERSION="v1.31.0"
ARCH="amd64"
curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz" | sudo tar -C $DOWNLOAD_DIR -xz

    DOWNLOAD_DIR="/usr/local/bin"
sudo tar -xvf /opt/myshared/crictl-v1.31.0-linux-amd64.tar.gz -C $DOWNLOAD_DIR

5. Install kubeadm, kubelet and add a kubelet systemd service:

RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
ARCH="amd64"
cd $DOWNLOAD_DIR
sudo curl -L --remote-name-all https://dl.k8s.io/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet}
sudo chmod +x {kubeadm,kubelet}

RELEASE_VERSION="v0.16.2"
curl -k -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubelet/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /usr/lib/systemd/system/kubelet.service
sudo mkdir -p /usr/lib/systemd/system/kubelet.service.d
curl -k -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

DOWNLOAD_DIR="/usr/local/bin"
sudo mkdir -p "$DOWNLOAD_DIR"
sudo cp /opt/myshared/kubeadm $DOWNLOAD_DIR
sudo cp /opt/myshared/kubelet $DOWNLOAD_DIR
sudo chmod +x "$DOWNLOAD_DIR"/{kubeadm,kubelet}

sudo install -o root -g root -m 0755 /opt/myshared/kubectl /usr/local/bin/kubectl
kubectl version --client

# open original file, change `/usr/bin` to `$DOWNLOAD_DIR`, and save it to `/usr/lib/systemd/system/kubelet.service`

sudo cat /opt/myshared/kubelet.service | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /usr/lib/systemd/system/kubelet.service

sudo mkdir -p /usr/lib/systemd/system/kubelet.service.d
sudo cat /opt/myshared/10-kubeadm.conf | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" | sudo tee /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

sudo systemctl enable --now kubelet

6. Install CNI plugins cni-plugins-linux-amd64-v1.3.0.tgz, github download link (required for most pod network) :

CNI_PLUGINS_VERSION="v1.3.0"
ARCH="amd64"
DEST="/opt/cni/bin"
sudo mkdir -p "$DEST"
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_VERSION}/cni-plugins-linux-${ARCH}-${CNI_PLUGINS_VERSION}.tgz" | sudo tar -C "$DEST" -xz

DEST="/opt/cni/bin"
sudo mkdir -p "$DEST"
sudo tar -xvf /opt/myshared/cni-plugins-linux-amd64-v1.3.0.tgz -C $DEST

7. Offline preparation (docs)

kubeadm config images list

osboxes@osboxes:/usr/local/bin$ kubeadm config images list
registry.k8s.io/kube-apiserver:v1.32.0
registry.k8s.io/kube-controller-manager:v1.32.0
registry.k8s.io/kube-scheduler:v1.32.0
registry.k8s.io/kube-proxy:v1.32.0
registry.k8s.io/coredns/coredns:v1.11.3
registry.k8s.io/pause:3.10
registry.k8s.io/etcd:3.5.16-0

download images directly to your server
kubeadm config images pull

# or downlaod images from another machine via docker
docker pull registry.k8s.io/kube-apiserver:v1.32.0
docker pull registry.k8s.io/kube-controller-manager:v1.32.0
docker pull registry.k8s.io/kube-scheduler:v1.32.0
docker pull registry.k8s.io/kube-proxy:v1.32.0
docker pull registry.k8s.io/coredns/coredns:v1.11.3
docker pull registry.k8s.io/pause:3.10
docker pull registry.k8s.io/etcd:3.5.16-0

# export these images to files
docker save registry.k8s.io/kube-apiserver:v1.32.0 > ./kube-apiserver_v1.32.0.tar
docker save registry.k8s.io/kube-controller-manager:v1.32.0 > ./kube-controller-manager_v1.32.0.tar
docker save registry.k8s.io/kube-scheduler:v1.32.0 > ./kube-scheduler_v1.32.0.tar
docker save registry.k8s.io/kube-proxy:v1.32.0 > ./kube-proxy_v1.32.0.tar
docker save registry.k8s.io/coredns/coredns:v1.11.3 > ./coredns_v1.11.3.tar
docker save registry.k8s.io/pause:3.10 > ./pause_3.10.tar
docker save registry.k8s.io/etcd:3.5.16-0 > ./etcd_3.5.16-0.tar

# copy the files to the server and import them. ctr command comes with containerd 
## crictl'nin load'u yokmuş :(
## ctr tool'unu kullanarak importu yapabiliriz (https://github.com/kubernetes-sigs/cri-tools/issues/546#issuecomment-646909445).
ctr image import /opt/myshared/offline-images/kube-apiserver_v1.32.0.tar
ctr image import /opt/myshared/offline-images/kube-controller-manager_v1.32.0.tar
ctr image import /opt/myshared/offline-images/kube-scheduler_v1.32.0.tar
ctr image import /opt/myshared/offline-images/kube-proxy_v1.32.0.tar
ctr image import /opt/myshared/offline-images/coredns/coredns_v1.11.3.tar
ctr image import /opt/myshared/offline-images/pause_3.10.tar
ctr image import /opt/myshared/offline-images/etcd_3.5.16-0.tar

# verify loaded images with : 
sudo kubeadm config images pull --v=5

8. Setup cluster via kubeadm (docs):

  • learn ip and route info:

    To find out what this IP is on a Linux host you can use (docs):

    # Learn ip 
ip a

    # Look for a line starting with "default via"
ip route show 

    sudo kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.0.2.4 --v=5

    you must install a CNI to complete the installation, otherwise kubelet service will restart indefinitely.

  • install Calico CNI(docs):

    • install the operator:

      kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.1/manifests/tigera-operator.yaml

    • download custom resources

      curl https://raw.githubusercontent.com/projectcalico/calico/v3.29.1/manifests/custom-resources.yaml -O

    • and modify cidr: 10.244.0.0/16 value according to the cluster init argument pod-network-cidr value:

      # change the cidr: 10.244.0.0/16 value according to the value you have given in your `kubeadm init --pod-network-cidr=X.X.X.X` command

nano custom-resources.yaml

    • give the manifest to kubectl:

      kubectl create -f custom-resources.yaml

    • watch the changes:

      watch kubectl get pods -n calico-system

    • install calicoctl(docs):

      curl -L https://github.com/projectcalico/calico/releases/download/v3.29.1/calicoctl-linux-amd64 -o calicoctl
chmod +x ./calicoctl
sudo mv calicoctl /usr/local/bin
calicoctl version
sudo calicoctl node status

  • check the status of the containers via these commands

watch sudo  crictl ps -a

# get node status
kubectl get nodes

osboxes@osboxes:~$ kubectl get nodes
NAME      STATUS     ROLES           AGE     VERSION
osboxes   NotReady   control-plane   5m28s   v1.32.0

# get every pod status
kubectl get pods -A

osboxes@osboxes:~$ kubectl get pods -A
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE  
kube-system   coredns-668d6bf9bc-2z27x          0/1     Pending   0          4m20s
kube-system   coredns-668d6bf9bc-btnpc          0/1     Pending   0          4m20s
kube-system   etcd-osboxes                      1/1     Running   0          4m31s
kube-system   kube-apiserver-osboxes            1/1     Running   0          4m28s
kube-system   kube-controller-manager-osboxes   1/1     Running   0          4m28s
kube-system   kube-proxy-7z4rz                  1/1     Running   0          4m21s
kube-system   kube-scheduler-osboxes            1/1     Running   0          4m28s

# any problems occur, reset the cluster initialization via this command
kubeadm reset

kubectl create deployment nginx --image=nginx:latest
kubectl expose deployment nginx --type=LoadBalancer --name=nginx-service --port=80 --target-port=80

kubectl get deployments
kubectl get services

# because of the master node taint, no other deployments will be run on the   
# control-plane node. If you remove the taint, deployment will be completed 
# and nginx will be run:

# remove the taint from all nodes
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

# add the taint for node named `osboxes`
# kubectl taint nodes <node-name> node-role.kubernetes.io/control-plane:NoSchedule
kubectl taint nodes osboxes node-role.kubernetes.io/control-plane:NoSchedule

9. Install Helm(docs):

curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm

USEFUL INFOS:

Docker image import:

docker load < /opt/myshared/offline-images/kube-apiserver_v1.32.0.tar
docker load < /opt/myshared/offline-images/kube-controller-manager_v1.32.0.tar
docker load < /opt/myshared/offline-images/kube-scheduler_v1.32.0.tar
docker load < /opt/myshared/offline-images/kube-proxy_v1.32.0.tar
docker load < /opt/myshared/offline-images/coredns/coredns_v1.11.3.tar
docker load < /opt/myshared/offline-images/pause_3.10.tar
docker load < /opt/myshared/offline-images/etcd_3.5.16-0.tar

https://discuss.kubernetes.io/t/kubernetes-installation-issue-with-kubeadm-init/26785/4

CERT PROBLEMS FIX:

any cert problems configure them as explained here

cd /opt/myshared/
cp mycompany.intra.crt /usr/share/ca-certificates/
dpkg-reconfigure ca-certificates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment