gistfile1.txt

Elasticache cutover


1. maintenance page up (wait for active processes to stop)

in hipchat:
prod maintenance start

2. ssh into conv and web instances to stop workers

ec2 instances Name prod-conversion

ec2 instances Name prod-web

cd /opt/hellofax
./lib/ops/hf_stop.sh

3. ssh into memstalk and perform a redis dump

ssh [email protected]

redis-cli 

BGSAVE

LASTSAVE

4. cp the redis dump to s3

https://console.aws.amazon.com/s3/home?region=us-east-1#&bucket=hal_tmp&prefix=

s3cmd put /mnt/redis/6379/dump.rdb s3://hal_tmp/

5. use the redis dump from s3 to seed the new redis master in elasticache

s3cmd get s3://hal_tmp/dump.rdb dump.rdb

Which machine(s) do I do this on?

6. point dns at elasticache

https://console.aws.amazon.com/route53/home?region=us-east-1#resource-record-sets:ZTA7774TF4BMP

prod-2015-09-08.dmpa3d.ng.0001.use1.cache.amazonaws.com

7. ensure connectivity from web/conv

telnet prod-2015-09-08.dmpa3d.ng.0001.use1.cache.amazonaws.com 6379

8. start workers

./lib/ops/hf_start.sh

9. remove maintenance page

prod maintenance stop

Maintenance page up

prod maintenance start

Stop services

Allow beanstalkd to clear. Usually this isn't an issue, but if beanstalkd has a large number of delayed tasks or a long queue of ready tasks, we need to stop beanstalkd and get the bin log.
beanstalkd:
beanstalkd is empty:
./lib/ops/hf_stop.sh # on conversion
beanstalkd is not going to empty in a reasonable time
service monit stop; service beanstalkd stop # on old memstalk
cp /var/lib/beanstalkd/binlog* ~/; scp ~/binlog* NEW.MEMSTALK.IP:~/ # on old memstalk
service monit stop; service beanstalkd stop # on new memstalk
cp ~/binlog* /var/lib/beanstalkd/; service beanstalkd start; service monit start
R53 - move memstalk dns entry to new public dns value. Coming soon to Hal, use web interface for now.
./lib/ops/hf_start.sh # on conversion, also coming soon to Hal