HA kubeadm Cluster - Stacked Control Plane

README.md

Configuring an HA kubeadm Cluster with Stacked Control Plane

An example Vagrantfile that uses vagrant-libvirt is provided to be able to complete the steps below. Modifications will need to be made to use a different Vagrant provider.

Stand up your vagrant cluster

cd <path to Vagrantfile>
vagrant up

Building the latest v1.11 packages (prior to v1.11 release)

This step assumes you are still in the same directory as above, so that the built packages are available to the Vagrant guests through the /vagrant shared directory.

git clone https://github.com/kubernetes/kubernetes
cd kubernetes
git checkout release-1.11
bazel build build/debs:debs
cd ../
cp kubernetes/bazel-bin/build/debs/*.deb ./

Bootstrapping the first control plane host

vagrant ssh cp1
sudo apt-get install /vagrant/{cri_tools,kubelet,kubectl,kubeadm}.deb

K8S_VERSION=v1.11.0-beta.2
IP_ADDR=$(ip addr show dev ens5 scope global | grep inet | awk '{print $2}' | cut -d '/' -f 1)
cat <<EOF > kubeadm.conf
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: ${K8S_VERSION}
apiServerExtraArgs:
  endpoint-reconciler-type: lease
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://${IP_ADDR}:2379"
      advertise-client-urls: "https://${IP_ADDR}:2379"
      listen-peer-urls: "https://${IP_ADDR}:2380"
      initial-advertise-peer-urls: "https://${IP_ADDR}:2380"
      initial-cluster: "cp1=https://${IP_ADDR}:2380"
    serverCertSANs:
      - cp1
      - ${IP_ADDR}
    peerCertSANs:
      - cp1
      - ${IP_ADDR}
EOF

sudo kubeadm init --config kubeadm.conf
mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

mkdir -p /vagrant/cluster-secrets/etcd
sudo cp /etc/kubernetes/pki/{ca,front-proxy-ca}.{crt,key} /vagrant/cluster-secrets/
sudo cp /etc/kubernetes/pki/sa.{key,pub} /vagrant/cluster-secrets/
sudo cp /etc/kubernetes/pki/etcd/ca.{crt,key} /vagrant/cluster-secrets/etcd/
sudo cp /etc/kubernetes/admin.conf /vagrant/cluster-secrets/

exit # exit the cp1 Vagrant host

Add the second control plane node

vagrant ssh cp2
sudo apt-get install /vagrant/{cri_tools,kubelet,kubectl,kubeadm}.deb

sudo mkdir -p /etc/kubernetes/pki/etcd
sudo cp /vagrant/cluster-secrets/etcd/ca.{crt,key} /etc/kubernetes/pki/etcd/
sudo cp /vagrant/cluster-secrets/sa.{key,pub} /etc/kubernetes/pki/
sudo cp /vagrant/cluster-secrets/{ca,front-proxy-ca}.{crt,key} /etc/kubernetes/pki/

CP1_IP_ADDR=$(getent hosts cp1 | awk '{print $1}')
K8S_VERSION=v1.11.0-beta.2
IP_ADDR=$(ip addr show dev ens5 scope global | grep inet | awk '{print $2}' | cut -d '/' -f 1)
cat <<EOF > kubeadm.conf
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: ${K8S_VERSION}
apiServerExtraArgs:
  endpoint-reconciler-type: lease
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://${IP_ADDR}:2379"
      advertise-client-urls: "https://${IP_ADDR}:2379"
      listen-peer-urls: "https://${IP_ADDR}:2380"
      initial-advertise-peer-urls: "https://${IP_ADDR}:2380"
      initial-cluster: "cp1=https://${CP1_IP_ADDR}:2380,cp2=https://${IP_ADDR}:2380"
      initial-cluster-state: existing
    serverCertSANs:
      - cp2
      - ${IP_ADDR}
    peerCertSANs:
      - cp2
      - ${IP_ADDR}
EOF

sudo kubeadm alpha phase certs all --config kubeadm.conf
sudo kubeadm alpha phase kubelet config write-to-disk --config kubeadm.conf
sudo kubeadm alpha phase kubelet write-env-file --config kubeadm.conf
sudo kubeadm alpha phase kubeconfig kubelet --config kubeadm.conf
sudo systemctl start kubelet

KUBECONFIG=/vagrant/cluster-secrets/admin.conf kubectl exec -n kube-system etcd-cp1 -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP1_IP_ADDR}:2379 member add cp2 https://${IP_ADDR}:2380
sudo kubeadm alpha phase etcd local --config kubeadm.conf

sudo kubeadm alpha phase kubeconfig all --config kubeadm.conf
sudo kubeadm alpha phase controlplane all --config kubeadm.conf
sudo kubeadm alpha phase mark-master --config kubeadm.conf

exit # exit the cp2 Vagrant host

Add the third control plane node

vagrant ssh cp3
sudo apt-get install /vagrant/{cri_tools,kubelet,kubectl,kubeadm}.deb

sudo mkdir -p /etc/kubernetes/pki/etcd
sudo cp /vagrant/cluster-secrets/etcd/ca.{crt,key} /etc/kubernetes/pki/etcd/
sudo cp /vagrant/cluster-secrets/sa.{key,pub} /etc/kubernetes/pki/
sudo cp /vagrant/cluster-secrets/{ca,front-proxy-ca}.{crt,key} /etc/kubernetes/pki/

CP1_IP_ADDR=$(getent hosts cp1 | awk '{print $1}')
CP2_IP_ADDR=$(getent hosts cp2 | awk '{print $1}')
K8S_VERSION=v1.11.0-beta.2
IP_ADDR=$(ip addr show dev ens5 scope global | grep inet | awk '{print $2}' | cut -d '/' -f 1)
cat <<EOF > kubeadm.conf
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: ${K8S_VERSION}
apiServerExtraArgs:
  endpoint-reconciler-type: lease
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://${IP_ADDR}:2379"
      advertise-client-urls: "https://${IP_ADDR}:2379"
      listen-peer-urls: "https://${IP_ADDR}:2380"
      initial-advertise-peer-urls: "https://${IP_ADDR}:2380"
      initial-cluster: "cp1=https://${CP1_IP_ADDR}:2380,cp2=https://${CP2_IP_ADDR}:2380,cp3=https://${IP_ADDR}:2380"
      initial-cluster-state: existing
    serverCertSANs:
      - cp3
      - ${IP_ADDR}
    peerCertSANs:
      - cp3
      - ${IP_ADDR}
EOF

sudo kubeadm alpha phase certs all --config kubeadm.conf
sudo kubeadm alpha phase kubelet config write-to-disk --config kubeadm.conf
sudo kubeadm alpha phase kubelet write-env-file --config kubeadm.conf
sudo kubeadm alpha phase kubeconfig kubelet --config kubeadm.conf
sudo systemctl start kubelet

KUBECONFIG=/vagrant/cluster-secrets/admin.conf kubectl exec -n kube-system etcd-cp1 -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://${CP1_IP_ADDR}:2379 member add cp3 https://${IP_ADDR}:2380
sudo kubeadm alpha phase etcd local --config kubeadm.conf

sudo kubeadm alpha phase kubeconfig all --config kubeadm.conf
sudo kubeadm alpha phase controlplane all --config kubeadm.conf
sudo kubeadm alpha phase mark-master --config kubeadm.conf

exit # exit the cp3 Vagrant host

FUTURE.md

Thoughts on future Kubeadm work required to streamline experience

Currently the kubeadm config and workflow is centered around a single control plane node and any number of worker nodes. This requires additional workarounds when trying to bootstrap and manage clusters with an HA control plane. I believe there is a path forward that would help streamline the effort required to stand up a fully managed HA control plane including etcd.

• Etcd Config Changes
  • Modify the local etcd config to have an additional parameter for exposing etcd ports or to expose the ports by default
  • Add a attribute to override the exposed advertised ip, preferably by choosing an interface to use for detection
• Make API Server Load Balancer config more explicit instead of just overriding the api config
  • Add new attribute to specify a load balancer, which will implicitly set the api config if not explicitly set
• Control Plane join/extend workflow
  • Add a new top level operation for adding additional control plane nodes
  • Add a new Config primitive for new control plane nodes, minimal since the existing config can be queried through the apiserver
    • Require an appropriate kubeconfig and/or existing API Server and credentials
    • Alternatively leverage cmd line options to provide these and/or prompt the user in the case of creds to mask
  • Workflow:
    • Query API Server for existing cluster config
    • If CA/Certs not overriden in cluster config, use kubectl exec/cp to pull default CA files, service account keypair, etc
    • Create any required certificates
    • If using local etcd, use kubeadm exec/etcdctl to query cluster config
    • Perform necessary kubelet bootstrapping
    • Use kubectl exec/etcdctl to add new etcd node
    • Lay down etcd static manifest
    • Lay down other control plane component static manifests
    • Mark node as master
  • Enrich cluster-stored kubeadm config with additional information for this host
• Upgrade workflow should not require the control plane hosts be performed in any particular order, instead if we store config separately for init host vs joined host the upgrade process should be smart enough to handle upgrading the hosts in any order.
• This init/extend/join workflow should be consumable by cluster-api

TODO.md

TODO

• Add note about transition from 1 etcd node to 2 causing a brief service interruption
• Add load balancer into mix
  • Figure out way to be able to set api.ControlPlaneEndpoint, but also optionally override for controlplane component config to simplify bootstrapping
• Remove need for shared directory with steps
• Handle upgrades
  • etcd config

Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "heptio/quickstart-ubuntu"
  config.vm.provider "libvirt" do |libvirt|
     libvirt.memory = 2048
  end

  config.vm.provision "shell", inline: <<-EOF
sudo apt-get update
sudo apt-get install -y curl software-properties-common
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-get update
EOF

  config.vm.define :cp1, primary: true do |cp|
    cp.vm.hostname = "cp1"
  end
  config.vm.define :cp2, primary: true do |cp|
    cp.vm.hostname = "cp2"
  end
  config.vm.define :cp3, primary: true do |cp|
    cp.vm.hostname = "cp3"
  end
end