Procedure to Replace Single Master with Master Discovery Set to Static

Master Replacement with Static List.md

Master Replacement with Static List Procedure

Objective: WIth this procedure, we want to replace node 172.31.1.23 with node 172.31.7.30. Also, it is required that we change exhibitor_zk_path to something unique in order for this procedure to work.

Begin

Configure the bootstrap node with the modified config.yaml that has a change to a new exhibitor_zk_path along with the master node that you want to replace for master_list

Important NOTE: The most important piece is exhibitor_storage_backend: zookeeper and exhibitor_zk_path: "/masterip-change-20170127". You must set exhibitor_zk_path to be something unique otherwise your cluster wont converge.

Previous Config

---
bootstrap_url: http://172.31.11.148:80
cluster_name: IP_ADDRESS_MASTER_REPLACE
exhibitor_storage_backend: zookeeper
exhibitor_zk_hosts: 172.31.4.24:2181
exhibitor_zk_path: "/"
log_directory: /genconf/logs
master_discovery: static
master_list:
   - 172.31.14.42
   - 172.31.9.219
   - 172.31.7.61
   - 172.31.6.139
   - 172.31.1.23
mesos_dns_ip_sources:
- mesos
- docker
- host
process_timeout: 120
resolvers:
- 8.8.8.8
roles: slave_public
ssh_key_path: /genconf/ssh-key
weights: slave_public=1

After Change

---
bootstrap_url: http://172.31.11.148:80
cluster_name: IP_ADDRESS_MASTER_REPLACE
exhibitor_storage_backend: zookeeper
exhibitor_zk_hosts: 172.31.4.24:2181
exhibitor_zk_path: "/masterip-change-20170127"
log_directory: /genconf/logs
master_discovery: static
master_list:
   - 172.31.14.42
   - 172.31.9.219
   - 172.31.7.61
   - 172.31.6.139
   - 172.31.7.30
mesos_dns_ip_sources:
- mesos
- docker
- host
process_timeout: 120
resolvers:
- 8.8.8.8
roles: slave_public
ssh_key_path: /genconf/ssh-key
weights: slave_public=1

On the bootstrap node, run these commands below

 # Generate new config files with updated master list
sudo bash dcos_generate_config.ee.sh --genconf

 # Remove the docker restart 
sed -i -e "s/systemctl restart systemd-journald//g" -e "s/systemctl restart docker//g" genconf/serve/dcos_install.sh 

# Setup Nginx
sudo docker run -d -p 80:80 -v $PWD/genconf/serve:/usr/share/nginx/html:ro nginx 

Shutdown Permanently or Uninstall Old Master Being Replaced

# Uninstall
sudo -i /opt/mesosphere/bin/pkgpanda uninstall
sudo rm -rf /opt/mesosphere /var/lib/mesosphere /etc/mesosphere /var/lib/zookeeper /var/lib/dcos/exhibitor/zookeeper /var/lib/mesos /var/lib/dcos  /run/dcos
sudo rm -rf /etc/profile.d/dcos.sh /etc/systemd/journald.conf.d/dcos.conf /etc/systemd/system/dcos-download.service /etc/systemd/system/dcos-link-env.service /etc/systemd/system/dcos-setup.service /etc/systemd/system/multi-user.target.wants/dcos-setup.service /etc/systemd/system/multi-user.target.wants/dcos.target 
sudo rm -fr /run/mesos /var/log/mesos /tmp/dcos
sudo systemctl daemon-reload
sudo reboot
# Completed

On all master perform the upgrade (in any order and/or simultaneously – quorum needs to be recreated)

Existing Masters

# Master Commands
curl -O http://172.31.11.148/dcos_install.sh
sudo -i /opt/mesosphere/bin/pkgpanda uninstall
sudo rm -rf /opt/mesosphere /etc/mesosphere 
sudo useradd --system --home-dir /opt/mesosphere --shell /sbin/nologin -c 'DCOS System User' dcos_exhibitor 
sudo chown -R dcos_exhibitor /var/lib/zookeeper
sudo bash dcos_install.sh -d master
# Complete

New Master

# Master Commands
curl -O http://172.31.11.148/dcos_install.sh
sudo useradd --system --home-dir /opt/mesosphere --shell /sbin/nologin -c 'DCOS System User' dcos_exhibitor 
sudo chown -R dcos_exhibitor /var/lib/zookeeper
sudo bash dcos_install.sh -d master
# Complete

For the agents launch a marathon job that will replace the static ips locally on each agent:

We will be changing the ip from 172.31.1.23 to 172.31.7.30 in our sed command here 's/172.31.1.23/172.31.7.30/g' and printing the result. Change the number of instances from 1000000 to the number of agents currently registered on the cluster.

{
  "id": "/update-master-ip",
  "cmd": "sudo cat /opt/mesosphere/packages/dcos-config--setup_*/etc/master_list ;sudo sed -i.bak  's/172.31.1.23/172.31.7.30/g' /opt/mesosphere/packages/dcos-config--setup_*/etc/master_list; sudo cat /opt/mesosphere/packages/dcos-config--setup_*/etc/master_list  ; sleep 10000000",
  "cpus": 0.01,
  "mem": 32,
  "disk": 0,
  "instances": 1000000,
  "acceptedResourceRoles": [
    "*",
    "slave_public"
  ],
 "constraints": [
    [
      "hostname",
      "UNIQUE"
    ]
  ]
}

Output

Registered executor on 172.31.10.255
Starting task update-master-ip.b95d58a5-e749-11e6-b02b-663fb0863958
Forked command at 18349
sh -c 'sudo cat /opt/mesosphere/packages/dcos-config--setup**/etc/master_list ;sudo sed -i.bak  's/172.31.1.23/172.31.7.30/g' /opt/mesosphere/packages/dcos-config--setup_*/etc/master_list; sudo cat /opt/mesosphere/packages/dcos-config--setup_*/etc/master*list  ; sleep 10000000'
<"172.31.14.42", "172.31.9.219", "172.31.7.61", "172.31.6.139", "172.31.1.23">
["172.31.14.42", "172.31.9.219", "172.31.7.61", "172.31.6.139", "172.31.7.30"]

Finish