readme.md

Fix agents on broken hostname/ip

Updating the Server

1. Pick a hostname
2. Setup DNS or /etc/hosts files so all agents can resolve the name
3. For HA: upgrade or reinstall helm chart with new hostname/certs. Docker Container: update contiainer with new certs (see upgrade instructions) or just use the default self-signed
4. Change the server-url setting to the new hostname (settings -> server-url)

Fixing Agents

Browse to the clusterregistrationtokens API endpoint for the cluster (global -> clusters -> ellipsis menu -> view in API) Example: https://jgreat-test-3.eng.rancher.space/v3/clusters/c-c4h74/clusterregistrationtokens

Find the insecure_command and node_command values.

insecureCommand: "curl --insecure -sfL https://172.31.18.95/v3/import/74fxp7bp7kkdqrchhctw8fhh4ndfr79p86wmqzjpbrt9xn9ljcxkll.yaml | kubectl apply -f -"
nodeCommand": "sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.0.8 --server https://172.31.18.95 --token 74fxp7bp7kkdqrchhctw8fhh4ndfr79p86wmqzjpbrt9xn9ljcxkll --ca-checksum 7015897e5d36c0dd09c891b249e8c884b0908f5e080b15cd9d86e3d7320802fe"

Update docker node agent

ssh into your system and run the nodeCommand - don't forget to add --worker, --controlplane, --etcd flags as appropriate for the node role.

sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.0.8 --server https://172.31.18.95 --token 74fxp7bp7kkdqrchhctw8fhh4ndfr79p86wmqzjpbrt9xn9ljcxkll --ca-checksum 7015897e5d36c0dd09c891b249e8c884b0908f5e080b15cd9d86e3d7320802fe --worker --etcd --controlplane

List the rancher_agent containers and remove any old running containers. (ignore the share-mnt container)

docker ps -a | grep rancher_agent | grep -v share-mnt

TODO:

Update k8s cluster/node agents.

ssh into a controlplane node and exec into any of the kube-apiserver containers.

docker exec -it kube-apiserver bash

Export the KUBECONFIG variable, pointing at the config file in /etc/kubernetes/ssl

export KUBECONFIG=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml

Run the insecureCommand to update the k8s node/cluster agents

curl --insecure -sfL https://172.31.18.95/v3/import/74fxp7bp7kkdqrchhctw8fhh4ndfr79p86wmqzjpbrt9xn9ljcxkll.yaml | kubectl apply -f -