Contrail Upgrade with minimal downtime

contrail-upgrade.txt

Contrail Upgrade procedure – Minimal downtime

Upgrade procedure to minimize downtime in the production network. Upgrade controller and compute separately. The procedure is based on fab upgrade_contrail.

1. Workaround for the bug (https://bugs.launchpad.net/juniperopenstack/+bug/1506634 )

Create policy to deny stop/start of services during upgrade

fab -R control -- 'echo "#! /bin/sh" > /usr/sbin/policy-rc.d'
fab -R control -- 'echo "exit 101" >> /usr/sbin/policy-rc.d'
fab -R control -- 'chmod 755 /usr/sbin/policy-rc.d'

fab -R collector -- 'echo "#! /bin/sh" > /usr/sbin/policy-rc.d'
fab -R collector -- 'echo "exit 101" >> /usr/sbin/policy-rc.d'
fab -R collector -- 'chmod 755 /usr/sbin/policy-rc.d'


2.Controller upgrade

Upgrade controller first.

2.1. package installation

dpkg -i contrail-install-packages_<Release>-<Build>~juno_all.deb
/opt/contrail/contrail_packages/setup.sh
cd /opt/contrail/utils/; fab pre_check
fab install_pkg_all:/root/contrail-install-packages_<Release>-<Build>~juno_all.deb

2.2. database, config, collector upgrade
  API is stopped during this operation.

fab stop_cfgm
fab stop_rabbitmq [If rabbitmq server is not running on contrail controller, this step is not required]
fab stop_collector

fab upgrade_database:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb

	#The above single command can be replaced with following multiple commands if one want to upgrade the nodes one by one.
fab upgrade_database_node: 2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb,root@<DATABASE_NODE1>
fab upgrade_database_node: 2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb, root@<DATABASE_NODE2>
fab upgrade_database_node:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb, root@<DATABASE_NODE3>

fab upgrade_config:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb

	#The above single command can be replaced with following multiple commands if one want to upgrade the nodes one by one.
fab upgrade_config_node:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb, root@<CONFIG_NODE1>
fab upgrade_config_node:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb,root@<CONFIG_NODE2>
fab upgrade_config_node:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb,root@<CONFIG_NODE2>

fab setup_rabbitmq_cluster [If rabbitmq server is not running on contrail controller, this step is not required]

fab setup_haproxy_config_node # This will re-provision /etc/haproxy/haproxy.cfg in ALL the config nodes, To avoid this we can skip this step and make necessary changes manually in /etc/haproxy/haproxt.cfg and restart haproxy, Definition of this task is at /opt/contrail/utils/fabfile/tasks/provision.py
fab restart_cfgm

fab upgrade_collector:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb

	#The above single command can be replaced with following multiple commands if one want to upgrade the nodes one by one.
fab upgrade_collector_node:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb, root@<COLLECTOR_NODE1>
fab upgrade_collector_node:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb, root@<COLLECTOR_NODE1>	
fab upgrade_collector_node:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb, root@<COLLECTOR_NODE1>


2.3. Control node upgrade
  Upgrading control node one by one. It keeps control plane working.

fab upgrade_control_node:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb,root@<CONTROL_NODE1>
fab upgrade_control_node:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb,root@<CONTROL_NODE2>
fab upgrade_control_node:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb,root@<CONTROL_NODE3>

2.4. webui upgrade

fab upgrade_webui:2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb


3. Compute Node upgrade
  Upgrading compute node one by one. It minimizes downtime.

3.1. Upgrade vRouter (on Build server)

fab upgrade_compute_node: 2.20,/root/contrail-install-packages_<Release>-<Build>~juno_all.deb,root@<COMPUTE_NODEx>
fab rmmod_vrouter_node: root@<COMPUTE_NODEx> # BELOW PATCH REQUIRED IN BUILD/FAB MACHINE. THIS PATCH IS NOT REQUIRED AFTER 3.0 RELEASE
fab restart_vrouter_node:root@<COMPUTE_NODEx>
fab  -H  root@<COMPUTE_NODEx> -- 'service nova-compute restart'

3.2. Note

Patch is needed for rmmod_router. Patch the below diff in the build host.
 
root@a2s42:/opt/contrail/utils/fabfile/tasks# diff -u misc.py.orig  misc.py
--- misc.py.orig        2015-09-17 07:12:09.388364558 -0700
+++ misc.py     2015-09-17 07:15:49.841723238 -0700
@@ -<Build>,14 +<Build>,8 @@
def rmmod_vrouter_node(*args):
     """Removes the vrouter kernal module in one compoute node."""
     for host_string in args:
-        if getattr(testbed, 'data', None) and host_string in testbed.data.keys():
-            with settings(host_string=host_string):
-                sudo("service supervisor-vrouter stop")
-                sudo("rmmod vrouter")
-                sudo("modprobe -r vrouter")
-                sudo("service supervisor-vrouter start")
-        else:
-            print "Managment and data interface are the same."
+        with settings(host_string=host_string):
+                sudo("service supervisor-vrouter stop; modprobe -r vrouter || rmmod vrouter; modprobe vrouter; service supervisor-vrouter start")


3.3. Remove the debian policy file

fab -R control -- 'rm -rf /usr/sbin/policy-rc.d'
fab -R collector--  'rm -rf /usr/sbin/policy-rc.d'