Contrail backup and restore

contrail-backup-restore.txt

Contrail database backup/restore procedure

This procedure can be used to backup contrail database (except the ContrailAnalytics keyspace) during upgrade from R2.20#64 to R2.22.1#5. Incase of database corruption during the upgrade process, the backed up database can be restored on the cluster. The below procedure can be used to backup/restore contrail database except the ContrailAnalytics keyspace.

1.1	Dropping ContrailAnalytics keyspace before upgrade

If the user prefers to drop the ContrailAnalytics data before upgrading the cluster, then the following can be done:


From FAB_NODE, execute “fab stop_collector”

On one of the database node, execute: (no need to execute this on all databse nodes)
	cqlsh <IP> 9160
	cqlsh> drop keyspace “ContrailAnalytics”;
In fab node, execute:

FAB_NODE# fab -R database – "rm –rf /var/lib/cassandra/data/ContrailAnalytics"

1.2	Backup Contrail Database 

TO Backup the database except ContrailAnalytics (Refer to section “Backup Contrail Database”)

The following nomenclature is used in this procedure:

FAB_NODE : The node from where the fabric commands are run.
BACKUP_SERVER : Server where the backed up data is stored
DATABASE1, DATABASE2, DATABASE3 : Contrail database node 1,2,3.

1.2.1	Backup Zookeeper data

FAB_NODE# fab stop_cfgm

On BACKUP_SERVER create:
mkdir -p <backup_path>/db1/var/lib/zookeeper/
mkdir -p <backup_path>/db2/var/lib/zookeeper/
mkdir -p <backup_path>/db3/var/lib/zookeeper/

DATABASE1# scp -r /var/lib/zookeeper/* root@<BACKUP_SERVER>:<backup_path>/db1/var/lib/zookeeper/
DATABASE2# scp -r /var/lib/zookeeper/* root@<BACKUP_SERVER>:<backup_path>/db2/var/lib/zookeeper/
DATABASE3# scp -r /var/lib/zookeeper/* root@<BACKUP_SERVER>:<backup_path>/db3/var/lib/zookeeper/

1.2.2	Create snapshot of the database:

Create database snapshot on all the database nodes using the following commands.

FAB_NODE# fab -R database -- "nodetool -h localhost -p 7199 snapshot config_db_uuid"
FAB_NODE# fab -R database -- "nodetool -h localhost -p 7199 snapshot to_bgp_keyspace"
FAB_NODE# fab -R database -- "nodetool -h localhost -p 7199 snapshot svc_monitor_keyspace"

1.2.3	Copy the snapshots to the Backup Server

1.2.3.1	Copy config_db_uuid snapshot to the backup server

DATABASE1# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/config_db_uuid/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done

On BACKUP_SERVER create:
mkdir -p <backup_path>/db1/<of all the directories listed during above step>

DATABASE1# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db1/var/lib/cassandra/data/config_db_uuid/$snapshot_dir/; fi; done

DATABASE2# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/config_db_uuid/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done

On BACKUP_SERVER create:
mkdir -p <backup_path>/db2/<of all the directories listed during above step>

DATABASE2# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db2/var/lib/cassandra/data/config_db_uuid/$snapshot_dir/; fi; done

DATABASE3# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/config_db_uuid/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done

On BACKUP_SERVER create:
mkdir -p <backup_path>/db3/<of all the directories listed during above step>

DATABASE3# cd /var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db3/var/lib/cassandra/data/config_db_uuid/$snapshot_dir/; fi; done

1.2.3.2	Copy to_bgp_keyspace snapshot to the backup server

DATABASE1# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/to_bgp_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done

On BACKUP_SERVER create:
mkdir -p <backup_path>/db1/<of all the directories listed during above step>

DATABASE1# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then  snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db1/var/lib/cassandra/data/to_bgp_keyspace/$snapshot_dir/; fi; done

DATABASE2# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/to_bgp_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done

On BACKUP_SERVER create:
mkdir -p <backup_path>/db2/<of all the directories listed during above step>

DATABASE2# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then  snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db2/var/lib/cassandra/data/to_bgp_keyspace/$snapshot_dir/; fi; done

DATABASE3# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/to_bgp_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done

On BACKUP_SERVER create:
mkdir -p <backup_path>/db3/<of all the directories listed during above step>

DATABASE3# cd /var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db3/var/lib/cassandra/data/to_bgp_keyspace/$snapshot_dir/; fi; done

1.2.3.3	Copy svc_monitor_keyspace snapshot to the backup server

DATABASE1# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/svc_monitor_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done
On BACKUP_SERVER create:
mkdir -p <backup_path>/db1/<of all the directories listed during above step>
DATABASE1# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db1/var/lib/cassandra/data/svc_monitor_keyspace/$snapshot_dir/; fi; done

DATABASE2# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/svc_monitor_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done

On BACKUP_SERVER create:
mkdir -p <backup_path>/db2/<of all the directories listed during above step>

DATABASE2# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db2/var/lib/cassandra/data/svc_monitor_keyspace/$snapshot_dir/; fi; done

DATABASE3# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then echo /var/lib/cassandra/data/svc_monitor_keyspace/$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1)/; fi; done

On BACKUP_SERVER create:
mkdir -p <backup_path>/db3/<of all the directories listed during above step>

DATABASE3# cd /var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots/" ]; then snapshot_dir=$dir/snapshots/$(ls -t $dir/snapshots/ | head -n1); scp -r $snapshot_dir/* root@<BACKUP_SERVER>:<backup_path>/db3/var/lib/cassandra/data/svc_monitor_keyspace/$snapshot_dir/; fi; done

1.2.3.4	Copy the database initial tokens

On BACKUP_SERVER create:
mkdir -p <backup_path>/db1/initial_tokens/
mkdir -p <backup_path>/db2/initial_tokens/
mkdir -p <backup_path>/db3/initial_tokens/

DATABASE1: nodetool ring | grep <DATABASE1_Internal_IP> | awk '{print $NF ","}' | xargs > /tmp/initial_tokens
DATABASE1: scp /tmp/initial_tokens root@<BACKUP_SERVER>:<backup_path>/db1/initial_tokens/
DATABASE2: nodetool ring | grep <DATABASE2_Internal_IP> | awk '{print $NF ","}' | xargs > /tmp/initial_tokens
DATABASE2: scp /tmp/initial_tokens root@<BACKUP_SERVER>:<backup_path>/db2/initial_tokens/
DATABASE3: nodetool ring | grep <DATABASE3_Internal_IP> | awk '{print $NF ","}' | xargs > /tmp/initial_tokens
DATABASE3: scp /tmp/initial_tokens root@<BACKUP_SERVER>:<backup_path>/db3/initial_tokens/

1.3	Restore Contrail Database

fab stop_cfgm
fab stop_database

1.3.1	Restore Zookeeper data

Copy the previously backed up corresponding /var/lib/zookeeper/* files from the  BACKUP_SERVER

FAB_NODE# fab -R database -- "chown -R zookeeper:zookeeper /var/lib/zookeeper/"

ON DATABASE1,2,3
Delete the following
/var/lib/cassandra/data/config_db_uuid/*
/var/lib/cassandra/data/to_bgp_keyspace/*
/var/lib/cassandra/data/svc_monitor_keyspace/*


1.3.2	Restore Cassandra data

1.3.2.1	Restore config_db_uuid keyspace

On BACKUP_SERVER
cd <backup_path>/db1/var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE1> "mkdir -p /var/lib/cassandra/data/config_db_uuid/$dir";  scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE1>:/var/lib/cassandra/data/config_db_uuid/$dir/; fi; done

cd <backup_path>/db2/var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE2> "mkdir -p /var/lib/cassandra/data/config_db_uuid/$dir";  scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE2>:/var/lib/cassandra/data/config_db_uuid/$dir/; fi; done

cd <backup_path>/db3/var/lib/cassandra/data/config_db_uuid/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE3> "mkdir -p /var/lib/cassandra/data/config_db_uuid/$dir";  scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE3>:/var/lib/cassandra/data/config_db_uuid/$dir/; fi; done

1.3.2.2	Restore to_bgp_keyspace keyspace

On BACKUP_SERVER
cd <backup_path>/db1/var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE1> "mkdir -p /var/lib/cassandra/data/to_bgp_keyspace/$dir";  scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE1>:/var/lib/cassandra/data/to_bgp_keyspace/$dir/; fi; done

cd <backup_path>/db2/var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE2> "mkdir -p /var/lib/cassandra/data/to_bgp_keyspace/$dir";  scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE2>:/var/lib/cassandra/data/to_bgp_keyspace/$dir/; fi; done

cd <backup_path>/db3/var/lib/cassandra/data/to_bgp_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE3> "mkdir -p /var/lib/cassandra/data/to_bgp_keyspace/$dir";  scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE3>:/var/lib/cassandra/data/to_bgp_keyspace/$dir/; fi; done

1.3.2.3	Restore svc_monitor_keyspace keyspace

On BACKUP_SERVER
cd <backup_path>/db1/var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE1> "mkdir -p /var/lib/cassandra/data/svc_monitor_keyspace/$dir";  scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE1>:/var/lib/cassandra/data/svc_monitor_keyspace/$dir/; fi; done

cd <backup_path>/db2/var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE2> "mkdir -p /var/lib/cassandra/data/svc_monitor_keyspace/$dir";  scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE2>:/var/lib/cassandra/data/svc_monitor_keyspace/$dir/; fi; done

cd <backup_path>/db3/var/lib/cassandra/data/svc_monitor_keyspace/; for dir in $(ls); do if [ -d "$dir/snapshots" ]; then ssh root@<DATABASE3> "mkdir -p /var/lib/cassandra/data/svc_monitor_keyspace/$dir";  scp -r $dir/snapshots/$(ls $dir/snapshots/ | head -n1)/* root@<DATABASE3>:/var/lib/cassandra/data/svc_monitor_keyspace/$dir/; fi; done

1.3.2.4	Restore initial_tokens

On each of database node, copy the corresponding initial_token from BACKUP_SERVER to /tmp/initial_tokens

DATABASE1: echo "initial_token: $(cat /tmp/initial_tokens)" >> /etc/cassandra/cassandra.yaml
DATABASE2: echo "initial_token: $(cat /tmp/initial_tokens)" >> /etc/cassandra/cassandra.yaml
DATABASE3: echo "initial_token: $(cat /tmp/initial_tokens)" >> /etc/cassandra/cassandra.yaml

FAB_NODE# fab -R database -- "chown -R cassandra:cassandra /var/lib/cassandra/"

1.3.2.5	Repair and verify database

FAB_NODE# fab start_database
FAB_NODE# fab verify_database
FAB_NODE: fab -R database -- "nodetool repair  -- config_db_uuid"
FAB_NODE: fab -R database -- "nodetool repair  -- to_bgp_keyspace"
FAB_NODE: fab -R database -- "nodetool repair  -- svc_monitor_keyspace"

FAB_NODE# fab restart_cfgm