patrick0057

Change Rancher 2.x server hostname

Credit for the information in this document goes to Superseb. I am just publishing it in an easy to follow gist for later use. Before starting on this guide ensure you have offline backups of etcd for your local Rancher cluster and all of your downstream clusters. Steps outlined in this document are unsupported, use at your own risk. I recommend performing steps in a test environment first.

Navigate to Global> Settings> then find server-url in the list, click the triple dot and then "Edit'. Change the server-url to your desired value.
Navigate to https://$server-url/v3/clusterregistrationtoken?clusterId=$CLUSTERID and grab the value from Data> insecureCommand>
- Example value:
```
curl --insecure -sfL https://$server-url/v3/import/2bdrqnkjzc7rbjsvg6j6dv9hgttmjgl84dw8tz775qkczq8qkkhh6t.yaml | kubectl apply -f -`
```

Quick gist for curling etcd metrics. There are better ways to get the metrics but I'm creating this gist anyway in case I need to reference this again later.

export etcd_endpoint=$(docker exec etcd netstat -lpna | grep \:2379 | grep tcp | grep LISTEN | tr -s " " | cut -d" " -f4)

{ for var in $(docker inspect --format '{{ .Config.Env }}' etcd | sed 's/[][]//g'); do
if [[ "$var" == *"ETCDCTL_CERT"* ]] || [[ "$var" == *"ETCDCTL_KEY"* ]]; then
export ${var}
fi
done }

Update self signed certificate on single install of Rancher 2.x

Download Rancher single tool on the server that is running your Rancher container:

curl -LO https://github.com/patrick0057/rancher-single-tool/raw/master/rancher-single-tool.sh

Run script so that it upgrades your installation (you can upgrade to the same version) and pass flags to indicate that you want to regenerate your self signed certificate. The most reliable way is to just specify all of your options on the command line but the script does have an easy to use automated system as well as shown in option b.

a. Specify all flags on command line, including any rancher options you had and docker options. Option -s is required for generating new 10 year self signed SSL certificates.

Deploy new cluster agent YAML

If you've recently made changes to your Rancher installation like updating the server URL or changing the Rancher installation SSL, then you will likely need to redeploy your cluster agent YAML files from the Rancher.

Create a local admin user for use with this tutorial. Without this, the script cannot login to get the new deployment file. You cannot use user accounts that are tied to third party authentication such as LDAP, Active Directory or GitHub to name a few.
Login to a single controlplane node of the cluster you need to redeploy your agent YAML to.

Download the script:

curl -LO https://github.com/patrick0057/cluster-agent-tool/raw/master/cluster-agent-tool.sh

wget https://github.com/patrick0057/cluster-agent-tool/raw/master/cluster-agent-tool.sh

Change Rancher 2.x server-url

Single server installation

During this tutorial it is recommended to use the rancher-single-tool for Rancher single server installations. It isn't required but it makes the process much easier. As a result this guide will be based on using that tool.

Download the rancher-single-tool to the node that is running your rancher server container.

   curl -LO https://github.com/patrick0057/rancher-single-tool/raw/master/rancher-single-tool.sh
   wget https://github.com/patrick0057/rancher-single-tool/raw/master/rancher-single-tool.sh

Major disaster preparation and recovery

In a perfect world our clusters would never experience a complete and total failure where data from all nodes is unrecoverable. Unfortunately this scenario is very possible and has happened before. In this article I will outline how to best prepare your environment for recovery in situations like this.

Situation: Employee A accidentally deletes all of the VM's for a production cluster after testing his latest script. How do you recover?

Option A: Keep VM snapshots of all of the nodes so that you can just restore them if they are deleted.

Option B: Manually bootstrap a new controlplane and etcd node to match one of the original nodes that were deleted.

In this article, I'm going to focus on Option B. In order to bootstrap a controlplane,etcd node, you will need an etcd snapshot, Kubernetes certificates and the runlike commands from the core Kubernetes components. If you prepare ahead of time for something like this, you can save a lot of time when it comes

etcd performance testing and optimization

If your etcd logs start showing messages like the following, your storage might be too slow for etcd or the server might be doing too much for etcd to operate properly.

2019-08-11 23:27:04.344948 W | etcdserver: read-only range request "key:\"/registry/services/specs/default/kubernetes\" " with result "range_response_count:1 size:293" took too long (1.530802357s) to execute

If you storage is really slow you will even see it throwing alerts in your monitoring system. What can you do the verify the performance of your storage? If the storage is is not performing correctly, how can you fix it? After researching this I found an IBM article that went over this extensively. Their findings on how to test were very helpful. The biggest factor is your storage latency. If it is not well below 10ms in the 99th percentile, you will see warnings in the etcd logs. We can test this with a tool called fio which I will outline below.

Testing etcd per

kube-apiserver restart loop

If the kube-apiserver is in a restart loop, it is possible that one of the etcd servers it is trying to connect to is no longer reachable. It should be able to just move on to the next etcd server but in some rare cases it does not. In those situations you need to remove the bad etcd servers from its startup options as shown below.

Get runlike command for kube-apiserverwith the following command:

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock axeal/runlike kube-apiserver

Example output:

	#!/bin/bash
	#Blank values must be filled in unless specified otherwise by preceding comment.
	#Name of the rancher environment you want to deploy your node to
	export RANCHER_ENVIRONMENT='Default'
	#Full path of your Rancher CLI executable
	export RANCHERCLI_PATH='/usr/local/bin/rancher'
	export MACHINE_DRIVER='azure'
	#Name of your Node
	export NAME=''
	#AZURE_ENVIRONMENT can be: AzureChinaCloud, AzureGermanCloud, AzurePublicCloud or AzureUSGovernmentCloud

	cmd/cloud-controller-manager/app/controllermanager.go: return c.ClientBuilder.ClientOrDie(serviceAccountName)
	cmd/cloud-controller-manager/app/options/options.go: c.VersionedClient = rootClientBuilder.ClientOrDie("shared-informers")
	cmd/kube-controller-manager/app/apps.go: ctx.ClientBuilder.ClientOrDie("daemon-set-controller"),
	cmd/kube-controller-manager/app/apps.go: ctx.ClientBuilder.ClientOrDie("statefulset-controller"),
	cmd/kube-controller-manager/app/apps.go: ctx.ClientBuilder.ClientOrDie("replicaset-controller"),
	cmd/kube-controller-manager/app/apps.go: ctx.ClientBuilder.ClientOrDie("deployment-controller"),
	cmd/kube-controller-manager/app/autoscaling.go: hpaClient := ctx.ClientBuilder.ClientOrDie("horizontal-pod-autoscaler")
	cmd/kube-controller-manager/app/autoscaling.go: hpaClient := ctx.ClientBuilder.ClientOrDie("horizontal-pod-autoscaler")
	cmd/kube-controller-manager/app/autoscaling.go: hpaClient := ctx.ClientBuilder.ClientOrDie