dkeightley

Task

In a disaster recovery scenario, the control plane and etcd nodes managed by Rancher in a downstream cluster may no longer be available or functioning. The cluster can be rebuilt by adding control plane and etcd nodes again, followed by restoring from an available snapshot.

Pre-requisites

A cluster built by Rancher v2.x or the Rancher Kubernetes Engine CLI (RKE)
Nodes to add to the cluster with control plane and etcd roles with adequate resources
An offline copy of a snapshot to be used as the recovery point, often stored in S3 or copied off node filesystems to a backup location

Install the ALB controller

I installed it via helm without IAM roles for Service Accounts (IRSA) configured for the EKS cluster, so the controller pods used the policy's attached to the underlying instance profile of the worker nodes.

https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/deploy/installation/#summary

Install Rancher

To deploy Rancher, I used the following helm install command, note using a values file can make this syntax clearer and easier.

Install

curl -sfL https://get.rke2.io | sh -
systemctl enable rke2-server.service
systemctl start rke2-server.service

Env setup

Using pv-migrate, prometheus monitoring data can be migrated between PV/PVCs when migrating to monitoring v2

This assumes persistent storage is used with monitoring (ie, a PV/PVC exists) and is intended only for cluster monitoring (not project monitoring).

Pre-work

Monitoring v1 apps (in Cluster Manager) should be disabled (Tools > Monitoring)
Ensure the monitoring v1 apps are uninstalled, more details here
Monitoring v2 (in Cluster Explorer) should be installed
Install pv-migrate, steps available here
Configure a kubeconfig for the cluster

Exec into the etcd container

RKE1

docker exec -it etcd sh

	#!/bin/sh

	PUBLIC_IP=$(curl ifconfig.io)

	# export INSTALL_RKE2_VERSION="v1.20.5+rke2r1"

	curl -sfL https://get.rke2.io \| sh -

	provider_id="$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)/$(curl -s http://169.254.169.254/latest/meta-data/instance-id)"

	apiVersion: install.istio.io/v1alpha1
	kind: IstioOperator
	spec:
	components:
	ingressGateways:
	- enabled: true
	name: istio-ingressgateway
	k8s:
	hpaSpec:
	maxReplicas: 10 # ... default 5

	# Calico Version v3.20.0
	# https://docs.projectcalico.org/releases#v3.20.0
	# This manifest includes the following component versions:
	# calico/ctl:v3.20.0

	apiVersion: v1
	kind: ServiceAccount
	metadata:
	name: calicoctl
	namespace: kube-system

	SERVICE=my-nginx
	NAMESPACE=default
	PORT=80

	for ingresspod in $(kubectl -n ingress-nginx get pods -l app=ingress-nginx --template '{{range.items}}{{.metadata.name}}{{"\n"}}{{end}}')
	do
	echo $ingresspod
	for svcep in $(kubectl -n $NAMESPACE get ep $SERVICE -o json \| jq -r '.subsets[].addresses[].ip')
	do
	echo "=> ${svcep}"

	#!/bin/sh

	PUBLIC_IP=$(curl ifconfig.io)

	echo "Installing K3S"
	# export INSTALL_K3S_VERSION="v1.19.5+k3s2"
	curl -sfL https://get.k3s.io \| sh -s - --tls-san ${PUBLIC_IP}

	echo "Downlading cert-manager CRDs"
	wget -q -P /var/lib/rancher/k3s/server/manifests/ https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml