Steps to add a new etcd to the OpenShift cluster

Install etcd on the "NEW_ETCD"

Do not start the etcd service

# yum install etcd-2.3.7-4.el7.x86_64

Add iptable rules on the "NEW_ETCD"

# systemctl enable iptables.service --now

# iptables -N OS_FIREWALL_ALLOW
# iptables -t filter -I INPUT -j OS_FIREWALL_ALLOW
# iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT
# iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT
# iptables-save

Remote over the the etcd host that was listed first in the Ansible install. We will call this the "ETCD_CA_HOST"

This host will have the directory /etc/etcd/ca/, this directory will contain the openssl configuration needed to sign the new etcd certificate we will be creating with the etcd CA.

Generate certificates from "ETCD_CA_HOST"

This steps have been taken from the Ansible Role etcd_server_certificates
- Set variables and working directory

# cd /etc/etcd
# pwd
/etc/etcd

# export NEW_ETCD="etcd-1.openshift.com"

# host $NEW_ETCD
etcd-1.openshift.com has address 192.168.0.15

# export CN=$NEW_ETCD
# export SAN="IP:192.168.0.15"
# export PREFIX="./generated_certs/etcd-$CN/"

- Create server.csr and server.crt

# openssl req -new -keyout ${PREFIX}server.key \
  -config ca/openssl.cnf \
  -out ${PREFIX}server.csr \
  -reqexts etcd_v3_req -batch -nodes \
  -subj /CN=$CN

# openssl ca -name etcd_ca -config ca/openssl.cnf \
  -out ${PREFIX}server.crt \
  -in ${PREFIX}server.csr \
  -extensions etcd_v3_ca_server -batch

- Create peer.csr and peer.crt

# openssl req -new -keyout ${PREFIX}peer.key \
  -config ca/openssl.cnf \
  -out ${PREFIX}peer.csr \
  -reqexts etcd_v3_req -batch -nodes \
  -subj /CN=$CN

# openssl ca -name etcd_ca -config ca/openssl.cnf \
  -out ${PREFIX}peer.crt \
  -in ${PREFIX}peer.csr \
  -extensions etcd_v3_ca_peer -batch

- Copy ca.crt and etcd.conf and archive the contents in the directory

# cp ca.crt ${PREFIX}
# cp etcd.conf ${PREFIX}
# tar -czvf ${PREFIX}${CN}.tgz -C ${PREFIX} .

- Transfer files over to the new etcd member

# scp ${PREFIX}${CN}.tgz  $CN:/etc/etcd/

While still on "ETCD_CA_HOST" we will add "NEW_ETCD" to the etcd cluster, take a backup of etcd then transfer to "NEW_ETCD".

Add "NEW_ETCD" to the cluster
- The ID and ETCD_* values outputted will be used in later steps

#export ETCD_CA_HOST="master-1.openshift.com"
#export NEW_ETCD="etcd-1.openshift.com"

# host $NEW_ETCD
etcd-1.openshift.com has address 192.168.0.15
# export NEW_ETCD_IP="192.168.0.15"

# etcdctl -C https://${ETCD_CA_HOST}:2379 --ca-file=/etc/etcd/ca.crt     --cert-file=/etc/etcd/peer.crt     --key-file=/etc/etcd/peer.key member add ${NEW_ETCD} https://${NEW_ETCD_IP}:2380
Added member named etcd-1.openshift.com with ID 328c30c625727e8 to cluster

ETCD_NAME="etcd-1.openshift.com"
ETCD_INITIAL_CLUSTER="etcd-1.openshift.com=https://192.168.0.15:2380,master-2.openshift.com=https://192.168.0.8:2380,master-0.openshift.com=https://192.168.0.9:2380,master-1.openshift.com=https://192.168.0.6:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

Take backup of etcd and transfer contents to "NEW_ETCD"

# export NODE_ID="328c30c625727e8"
# etcdctl backup --keep-cluster-id --node-id ${NODE_ID} --data-dir /var/lib/etcd --backup-dir /var/lib/etcd/$NEW_ETCD-backup
# tar -cvf $NEW_ETCD-backup.tar.gz -C /var/lib/etcd/$NEW_ETCD-backup/ . 
# scp $NEW_ETCD-backup.tar.gz $NEW_ETCD:/var/lib/etcd/

Remote to "NEW_ETCD" and extract data, set permissions, and make changes to etcd.conf file.

Extract data and set permissions

# ls  /etc/etcd/
etcd-1.openshift.com.tgz  etcd.conf

# tar -xf /etc/etcd/etcd-1.openshift.com.tgz -C /etc/etcd/ --overwrite
# chown etcd:etcd /etc/etcd/*    

# ls /var/lib/etcd/
etcd-1.openshift.com-backup.tar.gz

# rm -rf /var/lib/etcd/member
# tar -xf /var/lib/etcd/etcd-1.openshift.com-backup.tar.gz -C /var/lib/etcd/
# chown -R etcd:etcd /var/lib/etcd/

Make change to etcd.conf file.
- Using the values generated in the previous step when the member was added to the cluster. Replace:
  - ETCD_NAME
  - ETCD_INITIAL_CLUSTER
  - ETCD_INITIAL_CLUSTER_STATE
- Replace IP addresses with "NEW_ETCD" ip address for:
  - ETCD_LISTEN_PEER_URLS
  - ETCD_LISTEN_CLIENT_URLS
  - ETCD_INITIAL_ADVERTISE_PEER_URLS
  - ETCD_ADVERTISE_CLIENT_URLS

Start etcd on "NEW_ETCD"

# systemctl enable etcd --now

Verify the cluster is healthy.

etcdctl -C https://${ETCD_CA_HOST}:2379 --ca-file=/etc/etcd/ca.crt     --cert-file=/etc/etcd/peer.crt     --key-file=/etc/etcd/peer.key cluster-health

Add the member to the master-config.yaml on all the masters under "etcdClientInfo.urls"

rjhowe/OSE-New-ETCD.md