dbprod upgrade procedure.md

Prep:

• disable chef runs for 2-nsp-mel, 2-aws-syd
• disable harvesters
• disable firewall between 2-nsp-mel -> 2-aws-syd

2-nsp-mel:

• upgrade to 9.4 on 2-nsp-mel
• chef

2-aws-syd replica:

• snapshot 2-aws-syd and bring up replica VM
• upgrade to 9.4
• chef (remember IP address)

AWS console:

• cutover to new 2-aws-syd

  • re-allocate elastic IP
  • re-enable firewall

• re-enable chef

• remove temp firewall rule in NSP

I think the first thing you want to do is disable the firewall, so 2-nsp-mel starts storing the replication logs. Julian can probably clarify further.

Another thing we can do is stop harvesters while this is happening.

2-aws runs in replication mode. On the replicated server, If there's no cluster found, then chef should manage the connection to the primary server, and copy the databases (the so called basebackup) and then go into synchronization mode by copying the primary's log files.

If it was me doing this, I would upgrade 2-nsp first - so that the basebackup and log files will be in 9.4 binary standard. Then wipe out the 9.3 2-aws cluster, upgrade to 9.4 and try to provision to bring 2-aws into sync.

The only issue is that the chef code is buggy and no resources were applied to fixing it - so some intervention was necessary - at least when this upgrade was done in the past.