Skip to content

Instantly share code, notes, and snippets.

@canhnt

canhnt/rancher-incident.md

Last active December 20, 2018 21:13
Show Gist options
  • Save canhnt/92b2530bc737833f6e90d13d54115a04 to your computer and use it in GitHub Desktop.
Save canhnt/92b2530bc737833f6e90d13d54115a04 to your computer and use it in GitHub Desktop.
Download ZIP
Rancher incident checks
Raw
rancher-incident.md

Check numbers of templatecontent custom resource objects in the etcd DB:

etcdctl get  /registry/management.cattle.io/templatecontents --keys-only --prefix | sed '/^\s*$/d' | wc -l

A bug in Rancher prior v2.0.8 and v2.1.0 added a lot of html files in templatecontent objects, which fills up etcd db and may cause K8s master nodes unavailable.

Delete all templatecontent custom resource objects:

etcdctl del /registry/management.cattle.io/templatecontents/ --prefix

Recreate nginx-proxy container that hanlde traffic from kubelet to masters:

docker run -d --name nginx-proxy -p 127.0.0.1:6443:6443 \
 -e CP_HOSTS=10.165.214.97,10.165.214.98,10.165.211.10 \
 -e PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
 -e NGINX_VERSION=1.14.0 \
 --entrypoint=nginx-proxy rancher/rke-tools:v0.1.16  \
 --restart on-failure
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment