Offline Kubeadm install

kubeadm-install-offline.md

This Gist has moved

This Gist has moved to podder-ai/kubernetes-offline-install-guideline Please check here for latest updates.

---

Requirement

Work machine: A linux machine can access to internet

Prepare offline data (Require internet access)

Install yumdownloader

• Yumdownloader is a simple, yet useful command-line utility that downloads any RPM package along with all required dependencies in one go. Install Yumdownloader using the following command as root user.

yum install yum-utils

Download Docker installer

• Download all dependencies from by following the documentation https://docs.docker.com/engine/installation/linux/docker-ce/centos/#install-using-the-repository

mkdir docker-rpms
cd docker-rpms
yumdownloader --resolve yum-utils device-mapper-persistent-data lvm2
# add docker-ce repo
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
yumdownloader --resolve docker-ce

Save dockers images on work machine

Pull docker image

docker pull k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0       
docker pull k8s.gcr.io/kube-apiserver-amd64:v1.11.2 
docker pull k8s.gcr.io/kube-controller-manager-amd64:v1.11.2 
docker pull k8s.gcr.io/kube-proxy-amd64:v1.11.2 
docker pull k8s.gcr.io/kube-scheduler-amd64:v1.11.2 
docker pull weaveworks/weave-npc:2.4.0   
docker pull weaveworks/weave-kube:2.4.0   
docker pull k8s.gcr.io/coredns:1.1.3   
docker pull k8s.gcr.io/etcd-amd64:3.2.18  
docker pull k8s.gcr.io/pause:3.1  

Save docker images to tarball.

Using save-load-docker-image.sh in comment below. save-load-docker-image.sh

./save-load-docker-image.sh save -d docker-images
Saving k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0 ...
Saving k8s.gcr.io/kube-controller-manager-amd64:v1.11.2 ...
Saving k8s.gcr.io/kube-proxy-amd64:v1.11.2 ...
Saving k8s.gcr.io/kube-apiserver-amd64:v1.11.2 ...
Saving k8s.gcr.io/kube-scheduler-amd64:v1.11.2 ...
Saving weaveworks/weave-npc:2.4.0 ...
Saving weaveworks/weave-kube:2.4.0 ...
Saving hello-world:latest ...
Saving k8s.gcr.io/coredns:1.1.3 ...
Saving k8s.gcr.io/etcd-amd64:3.2.18 ...
Saving k8s.gcr.io/pause:3.1 ...

Download kubernetes rpms

mkdir k8s-rpms
cd k8s-rpms
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
EOF
setenforce 0
yumdownloader --resolve kubelet kubeadm kubectl --disableexcludes=kubernetes

Download YAML templates

Download weave plugin configuration

curl -o weave-kube.yaml "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

Download kubernetes dashboard

curl -o kubernetes-dashboard.yaml https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

Installation (Not require internet access)

On master and nodes

Copy Docker rpms to the remote server

scp <docker_folder_with_rpms>/*.rpm <user>@<server>:<path>/<to>/<remote>/<folder>

Install Docker tools

cd docker-rpms
yum install -y *.rpm
systemctl enable docker && systemctl start docker

Ensure docker is started

systemctl status docker

In case docker is not started

systemctl enable docker && systemctl start docker

Copy docker images tarball to the remote server

scp <folder_with_images>/*.tar <user>@<server>:<path>/<to>/<remote>/<folder>

Load docker images on remote server

Using save-load-docker-image.sh in comment below. save-load-docker-image.sh

./save-load-docker-image.sh load -d docker-images
Loading ./k8s.gcr.io/kubernetes-dashboard-amd64.v1.10.0.tar ...
Loaded image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.0
Loading ./k8s.gcr.io/kube-controller-manager-amd64.v1.11.2.tar ...
Loaded image: k8s.gcr.io/kube-controller-manager-amd64:v1.11.2
Loading ./k8s.gcr.io/kube-proxy-amd64.v1.11.2.tar ...
Loaded image: k8s.gcr.io/kube-proxy-amd64:v1.11.2
Loading ./k8s.gcr.io/kube-apiserver-amd64.v1.11.2.tar ...
Loaded image: k8s.gcr.io/kube-apiserver-amd64:v1.11.2
Loading ./k8s.gcr.io/kube-scheduler-amd64.v1.11.2.tar ...
Loaded image: k8s.gcr.io/kube-scheduler-amd64:v1.11.2
Loading ./k8s.gcr.io/coredns.1.1.3.tar ...
Loaded image: k8s.gcr.io/coredns:1.1.3
Loading ./k8s.gcr.io/etcd-amd64.3.2.18.tar ...
Loaded image: k8s.gcr.io/etcd-amd64:3.2.18
Loading ./k8s.gcr.io/pause.3.1.tar ...
Loaded image: k8s.gcr.io/pause:3.1
Loading ./weaveworks/weave-npc.2.4.0.tar ...
Loaded image: weaveworks/weave-npc:2.4.0
Loading ./weaveworks/weave-kube.2.4.0.tar ...
Loaded image: weaveworks/weave-kube:2.4.0
Loading ./hello-world.latest.tar ...
Loaded image: hello-world:latest

For Centos7, enable sysctl configuration

Some users on RHEL/CentOS 7 have reported issues with traffic being routed incorrectly due to iptables being bypassed. You should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctl config, e.g.

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl --system

Copy kubernetes rpms to the remote server

scp <folder_with_rpms>/*.rpm <user>@<server>:<path>/<to>/<remote>/<folder>

Install kubernetes tools

cd k8s-rpms
yum install -y *.rpm
systemctl enable kubelet && systemctl start kubelet

On master

Kubeadm installation

Follow instruction from https://kubernetes.io/docs/getting-started-guides/kubeadm/

initalization

kubeadm init

Configure kubectl

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get nodes

Pods network plugin configuration

https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network

Copy weave plugin configuration

scp <folder_with_weave_yml>/weave-kube.yml <user>@<server>:<path>/<to>/<remote>/<folder>

Apply plugin

kubectl apply -f <folder_with_weave_yml>/weave-kube.yml

Master Isolation

By default, your cluster will not schedule pods on the master for security reasons. If you want to be able to schedule pods on the master, e.g. for a single-machine Kubernetes cluster for development, run:

kubectl taint nodes --all node-role.kubernetes.io/master-

https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#master-isolation

Kubernetes dashboard

https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

Copy dashboard deployment configuration

scp <folder_with_dashboard_yml>/kubernetes-dashboard.yml <user>@<server>:<path>/<to>/<remote>/<folder>

Apply dashboard deployment

kubectl apply -f <folder_with_dashboard_yml>/kubernetes-dashboard.yml
kubectl proxy
https://<master-ip>:<apiserver-port>/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

Fix kube-proxy problem

kubectl -n kube-system get ds -l "component=kube-proxy" -o json | jq ".items[0].spec.template.spec.containers[0].command |= .+ [\"--proxy-mode=userspace\"]" | kubectl apply -f - && kubectl -n kube-system delete pods -l "component=kube-proxy" 

On Node

kubeadm join --token <token> <master-ip>

Download Docker CE rpms

# download-docker-ce.sh
#! /bin/bash
yumdownloader --resolve yum-utils device-mapper-persistent-data lvm2
# add docker-ce repo
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
yumdownloader --resolve docker-ce

Save & load docker images

save-load-docker-image.sh

#!/usr/bin/env bash

# Script to (selectively) save/load multiple Docker images to/from a directory.
# Run ./save-load-docker-images.sh for help.

set -e

directory=$PWD
filter=""
compress=0

while getopts ":f:d:z" opt ${@:2}; do
  case $opt in
    f)
      filter=$OPTARG
      ;;
    d)
      directory=$OPTARG
      ;;
    z)
      compress=1
      ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      ;;
  esac
done

help () {
    echo
    echo "\
Usage: save [-f filter] [-d dir]   Save all Docker images to a directory
       load [-f filter] [-d dir]   Find all saved images (.tar) in a directory then import to Docker

       -d dir                     Directory to save/load images to/from (defaults to \$PWD)
       -f filter                  Filter images by their name (inclusive)
       -z                         Use gzip to compress/uncompress archives (saved/loaded as *.tar.gz)"
    echo
}

get-image-field() {
  local imageId=$1
  local field=$2
  : ${imageId:? required}
  : ${field:? required}

  docker images --no-trunc | sed -n "/${imageId}/s/  */ /gp" | cut -d " " -f $field
}

get-image-name() {
  get-image-field $1 1
}

get-image-tag() {
  get-image-field $1 2
}

save-all-images() {
  local ids=$(docker images --no-trunc -q)
  local name safename tag

  for id in $ids; do
    name=$(get-image-name $id)
    tag=$(get-image-tag $id)

    # Apply filter (if any)
    if [[ ! -z "$filter" ]] && [[ ! "$name:$tag" =~ "$filter" ]];then
      continue
    fi

    # Ignore stale images (tag == <none>)
    if [[ "$tag" = "<none>" ]]; then
      continue
    fi

    if [[ $name =~ / ]]; then
       dir=${name%/*}
       mkdir -p "$directory/$dir"
    fi

    echo "Saving $name:$tag ..."
    if [[ $compress -eq 0 ]]; then
       docker save -o "$directory/$name.$tag.tar" $name:$tag
    else
      docker save $name:$tag | gzip > "$directory/$name.$tag.tar.gz"
    fi
  done
}

load-all-images() {
  local name safename noextension tag

  if [[ $compress -eq 0 ]]; then
    file_extension="tar"
  else
    file_extension="tar.gz"
  fi
  for image in $(find "$directory" -name \*.$file_extension); do
    if [[ ! -z "$filter" ]] && [[ ! "$image" =~ "$filter" ]];then
      continue
    fi
    echo "Loading $image ..."
    docker load -i $image
  done
}

case $1 in
    save)
      save-all-images
    ;;
    load)
      load-all-images
    ;;
    *)
        help
    ;;
esac

exit 0