Contrail Upgrade with minimal downtime (without fab)

Contrail Upgrade procedure – Minimal downtime

WARNING: This procedure only works for upgrades between Contrail versions PRIOR TO 3.0. Starting with 3.0, Cassandra version changes and this involves more work.

NOTE: This procedure is *UNSUPPORTED* because fab adds much more error-checking logic than what you can see here. Use this procedure at your own risk.

This document is inspired on another one ( https://gist.github.com/dsundarraj/9dde07f28208737c7729 ), but it breaks the fab steps into commands run on each of the nodes separately.

Upgrade procedure to minimize downtime in the production network. Upgrade controller and compute separately. The procedure is based on fab upgrade_contrail.

1. Workaround for the bug (https://bugs.launchpad.net/juniperopenstack/+bug/1506634 )

Create policy to deny stop/start of services during upgrade.

On all configuration, analytics nodes:

echo "#! /bin/sh" > /usr/sbin/policy-rc.d
echo "exit 101" >> /usr/sbin/policy-rc.d
chmod 755 /usr/sbin/policy-rc.d

2. Transfer the image to /root of all servers (control, configuration, analytics, database, compute).

Then install contrail-install-packages and create the repo:

dpkg -i /root/contrail-install-packages_<Release>-<Build>~<OpenStack>_all.deb
/opt/contrail/contrail_packages/setup.sh

You may want to use a manual method to extract the necessary packages and create the repo in a customized manner for each server.

3. Stop the database clients and the message broker:

On all the configuration nodes:

service supervisor-config stop
service neutron-server stop

On all the Contrail rabbitmq nodes:

service supervisor-support-service stop

On all the analytics nodes:

service supervisor-analytics stop

4. Upgrade the database nodes.

On all the database nodes:

ln -sf /bin/true /sbin/chkconfig
mkdir /var/tmp/contrail-upgradesave
mkdir /var/tmp/contrail-upgradesave/etc
cp -rfp /etc/contrail /var/tmp/contrail-upgradesave/etc/
DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-database
cp -rfp /var/tmp/contrail-upgradesave/etc/contrail/contrail-database-nodemgr.conf /etc/contrail/contrail-database-nodemgr.conf
service zookeeper restart
service supervisor-database restart

5. Upgrade the config nodes.

On all the config nodes:

ln -sf /bin/true /sbin/chkconfig
mkdir /var/tmp/contrail-upgradesave
mkdir /var/tmp/contrail-upgradesave/etc
mkdir /var/tmp/contrail-upgradesave/etc/init.d
mkdir /var/tmp/contrail-upgradesave/etc/sudoers.d

cp -rfp /etc/contrail /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/ifmap-server /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/neutron /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/init.d/contrail-api /var/tmp/contrail-upgradesave/etc/init.d/
cp -rfp /etc/init.d/contrail-discovery /var/tmp/contrail-upgradesave/etc/init.d/
cp -rfp /etc/sudoers.d/contrail_sudoers /var/tmp/contrail-upgradesave/etc/sudoers.d/

DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-config

cp -rfp /var/tmp/contrail-upgradesave/etc/contrail /etc/
cp -rfp /var/tmp/contrail-upgradesave/etc/ifmap-server /etc/
cp -rfp /var/tmp/contrail-upgradesave/etc/neutron /etc/
cp -rfp /var/tmp/contrail-upgradesave/etc/init.d/contrail-api /etc/init.d/
cp -rfp /var/tmp/contrail-upgradesave/etc/init.d/contrail-discovery /etc/init.d/
cp -rfp /var/tmp/contrail-upgradesave/etc/sudoers.d/contrail_sudoers /etc/sudoers.d/ 

On all the Contrail rabbitmq nodes:

service supervisor-support-service restart

On all the configuration nodes:

service supervisor-config restart
service neutron-server restart

6. Upgrade the analytics nodes.

On all the analytics nodes:

mkdir /var/tmp/contrail-upgradesave
mkdir /var/tmp/contrail-upgradesave/etc
cp -rfp /etc/contrail /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/redis /var/tmp/contrail-upgradesave/etc/

DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-analytics

cp -rfp /var/tmp/contrail-upgradesave/etc/contrail /etc/

sudo service redis-server restart
service supervisor-analytics restart

7. Upgrade the control nodes one by one, so there is always at least one BGP session up to the gateway and XMPP sessions to the compute nodes.

On each control node:

mkdir /var/tmp/contrail-upgradesave
mkdir /var/tmp/contrail-upgradesave/etc
cp -rfp /etc/contrail /var/tmp/contrail-upgradesave/etc/

DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-control

cp -rfp /var/tmp/contrail-upgradesave/etc/contrail /etc/

service supervisor-control restart

The BGP and XMPP sessions on this control node flaps.

8. Upgrade the webui nodes.

On all the webui nodes:

DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-openstack-webui

cp /var/tmp/contrail-upgradesave/etc/contrail/config.global.js /etc/contrail/
cp /var/tmp/contrail-upgradesave/etc/contrail/contrail-webui-userauth.js /etc/contrail/
service redis-server restart
service supervisor-webui restart

9. Upgrade the compute nodes.

On all the compute nodes:

mkdir /var/tmp/contrail-upgradesave
mkdir /var/tmp/contrail-upgradesave/etc
cp -rfp /etc/nova /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/contrail /var/tmp/contrail-upgradesave/etc/
cp -rfp /etc/libvirt /var/tmp/contrail-upgradesave/etc/

ln -sf /bin/true /sbin/chkconfig
DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-overwrite" -o Dpkg::Options::="--force-confnew" install contrail-vrouter-3.13.0-40-generic contrail-openstack-vrouter

cp -rfp /var/tmp/contrail-upgradesave/etc/contrail /etc/
cp -rfp /var/tmp/contrail-upgradesave/etc/nova /etc/
cp -rfp /var/tmp/contrail-upgradesave/etc/libvirt /etc/

service supervisor-vrouter stop; modprobe -r vrouter || rmmod vrouter; modprobe vrouter; service supervisor-vrouter start
service nova-compute restart

10. Remove workaround for the bug (https://bugs.launchpad.net/juniperopenstack/+bug/1506634 )

On all configuration, analytics nodes:

rm /usr/sbin/policy-rc.d