kube-apiserver restart loop

If the kube-apiserver is in a restart loop, it is possible that one of the etcd servers it is trying to connect to is no longer reachable. It should be able to just move on to the next etcd server but in some rare cases it does not. In those situations you need to remove the bad etcd servers from its startup options as shown below.

Get runlike command for kube-apiserverwith the following command:

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock axeal/runlike kube-apiserver

Example output:

   [root@ip-172-31-14-184 ~]# docker run --rm -v /var/run/docker.sock:/var/run/docker.sock axeal/runlike kube-apiserver
   Unable to find image 'axeal/runlike:latest' locally
   latest: Pulling from axeal/runlike
   bdf0201b3a05: Pull complete
   38f114998adb: Pull complete
   21134b1a9e68: Pull complete
   1826484332dc: Pull complete
   a124bc45399b: Pull complete
   e1147593372b: Pull complete
   2bd42cd7a205: Pull complete
   92dd7b6abd56: Pull complete
   ff342074ab25: Pull complete
   c96b6f85e13d: Pull complete
   c6f85279bce2: Pull complete
   69240e7fa944: Pull complete
   c2cab7570ce0: Pull complete
   Digest: sha256:8039674ccacc1ddb851723e49919ff36896d078b35067c8e80462064aacd0068
   Status: Downloaded newer image for axeal/runlike:latest
   docker run --name=kube-apiserver --hostname=ip-172-31-14-184.us-east-2.compute.internal --env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin --volume=/etc/kubernetes:/etc/kubernetes:z --volumes-from=service-sidekick --network=host --restart=always --label org.label-schema.schema-version="1.0" --label io.rancher.rke.container.name="kube-apiserver" --label org.label-schema.vcs-ref="c5da50e996b51071c14ffe3a83bd0f4ab469b3e0" --label org.label-schema.build-date="2019-06-12T03:01:55Z" --label org.label-schema.vcs-url="https://github.com/rancher/hyperkube.git" --entrypoint /opt/rke-tools/entrypoint.sh --detach=true rancher/hyperkube:v1.14.3-rancher1 kube-apiserver --requestheader-username-headers=X-Remote-User --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-lookup=true --storage-backend=etcd3 --etcd-prefix=/registry --etcd-servers=https://172.31.14.184:2379,https://172.31.4.50:2379,https://172.31.5.114:2379 --bind-address=0.0.0.0 --insecure-port=0 --profiling=false --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-node-port-range=30000-32767 --authorization-mode=Node,RBAC --anonymous-auth=false --requestheader-allowed-names=kube-apiserver-proxy-client --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --service-cluster-ip-range=10.43.0.0/16 --allow-privileged=true --requestheader-group-headers=X-Remote-Group --cloud-provider= --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --secure-port=6443 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.14.184 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-extra-headers-prefix=X-Remote-Extra-

Find the option that lists your etcd servers

--etcd-servers=https://172.31.14.184:2379,https://172.31.4.50:2379,https://172.31.5.114:2379

Remove the dead etcd server(s), delete your old kube-apiserver container then create a new one with the modified runlike command.

patrick0057/README.md

kube-apiserver restart loop