Removing and Adding Back etcd memember.

gistfile1.md

CONTEXT: 3 Masters:

master1.openshift.com 172.17.28.10
master2.openshift.com 172.17.28.12
master3.openshift.com 172.17.28.18

In this example we will be adding "master2.openshift.com" back into the cluster after it was removed.

STEPS:

1. Ensure etcd is updated on all etcd hosts and using golang 6

    ┌─[root@master1]─[~]
    └──> etcd --version
    etcd Version: 2.2.5
    Git SHA: bc9ddf2
    Go Version: go 1.6.2
    Go OS/Arch: linux/amd64
   

2. Checking cluster and stop etcd on master2

┌─[root@master1]─[~]
└──> etcdctl -C     https://master1.openshift.com:2379   --ca-file=/etc/origin/master/master.etcd-ca.crt     --cert-file=/etc/origin/master/master.etcd-client.crt     --key-file=/etc/origin/master/master.etcd-client.key member list
1e49ac32bec28074: name=master1.openshift.com peerURLs=https://172.17.28.10:2380 clientURLs=https://172.17.28.10:2379
40ae3b88f5f6f8zz: name=master2.openshift.com peerURLs=https://172.17.28.12:2380 clientURLs=https://172.17.28.12:2379
f478013b5b269dc9: name=master3.openshift.com peerURLs=https://172.17.28.18:2380 clientURLs=https://172.17.28.18:2379

1. Removing the member

   • Stop etcd on master2

   ┌──[root@master1]─[~]
   └──> ssh master2 systemctl stop etcd 
   

• Remove the memeber from the cluster

┌─[root@master1]─[~]
└──> etcdctl -C https://master1.openshift.com:2379,https://master2.openshift.com:2379,https://master3.openshift.com:2379     --ca-file=/etc/origin/master/master.etcd-ca.crt     --cert-file=/etc/origin/master/master.etcd-client.crt     --key-file=/etc/origin/master/master.etcd-client.key member remove 40ae3b88f5f6f8zz

1. Adding member "master2.openshift.com"

   • After running the command the output displayed will need to be added to the ectd.conf file on "master2.openshift.com"

   ┌─[root@master1]─[~]
   └──>  etcdctl -C     https://master1.openshift.com:2379,https://master2.openshift.com:2379,https://master3.openshift.com:2379     --ca-file=/etc/origin/master/master.etcd-ca.crt     --cert-file=/etc/etcd/peer.crt     --key-file=/etc/etcd/peer.key member add master2.openshift.com https://172.17.28.12:2380
   
   ETCD_NAME="master2.openshift.com"
   ETCD_INITIAL_CLUSTER="master1.openshift.com=https://172.17.28.10:2380,master2.openshift.com=https://172.17.28.12:2380,master3.openshift.com=https://172.17.28.18:2380"
   ETCD_INITIAL_CLUSTER_STATE="existing"
   

2. Remote to new etcd member master2.openshift.com. In our case this member is master2.openshift.com and will already have its previous config setup from initial install.

   • Make sure data-dir is removed and does not have old data.

   ┌──[root@master2]─[~]
   └──> rm -rf /var/lib/etcd/member
   

   • Add the following data to etcd.conf, you may need to update these fields as some information will already be there. Make sure the file shows the variable set correctly according to the output we got from adding the member. All that might need to be changed is ETCD_INITIAL_CLUSTER_STATE value, from "new" to "existing"

   ┌──[root@master2]─[~]
   └──> vi /etc/etcd/etcd.conf
   ...
   ... 
   ETCD_NAME="master2.openshift.com"
   ETCD_INITIAL_CLUSTER="master1.openshift.com=https://172.17.28.10:2380,master2.openshift.com=https://172.17.28.12:2380,master3.openshift.com=https://172.17.28.18:2380"
   ETCD_INITIAL_CLUSTER_STATE="existing"
   ...
   ...
   

   • Start the etcd service.

   ┌──[root@master2]─[~]
   └──> systemctl start etcd
   

3. Check the health of the cluster.

   ┌─[root@master1]─[~]
   └──> etcdctl -C     https://master1.openshift.com:2379   --ca-file=/etc/origin/master/master.etcd-ca.crt     --cert-file=/etc/origin/master/master.etcd-client.crt     --key-file=/etc/origin/master/master.etcd-client.key member list
   
   member 1e49ac32bec28074 is healthy: got healthy result from https://172.17.28.10:2379
   member 30ae3b88f5f6f8aa is healthy: got healthy result from https://172.17.28.12:2379
   member f478013b5b269dc9 is healthy: got healthy result from https://172.17.28.18:2379
   cluster is healthy
   

4. Add the member back to the master-config.yaml on all the masters.

   ┌─[root@master1]─[~]
   └──> vi /etc/origin/master/master-config.yaml
   
    etcdClientInfo:
      ca: master.etcd-ca.crt
      certFile: master.etcd-client.crt
      keyFile: master.etcd-client.key
      urls:
        - https://master1.openshift.com:2379
        - https://master2.openshift.com:2379
        - https://master3.openshift.com:2379
   
   ┌─[root@master1]─[~]
   └──> systemctl restart atomic-openshift-master-*