Testing moving cassandra from CFN to bare

1.Find and remove the first node

 ```
 $ kcs s  create Cassandra  CassandraPriamCluster.template PN  ACount=2 DCount=2 ECount=2 InstanceCount=1
 ```
 
 The resulting cluster looks something like this:
 ```
 (python)7235 pn@pn-mbp 11:44 ~/dvcs/github/chef-repo/environments $ hss s pn-cassandra -terminated
0. running	2014-01-31T16:27	i-b281df92	ec2-184-72-147-62.compute-1.amazonaws.com	Kerberos	cassandra_pn_kcs,base,pn_kcs,pn-cassandra,pn-cassandra
1. running	2014-01-31T16:27	i-b181df91	ec2-54-226-107-205.compute-1.amazonaws.com	Kerberos	cassandra_pn_kcs,base,pn_kcs,pn-cassandra
2. running	2014-01-31T16:24	i-d0bd21f1	ec2-184-72-73-37.compute-1.amazonaws.com	Kerberos	cassandra_pn_kcs,base,pn_kcs,pn-cassandra,pn-cassandra
3. running	2014-01-31T16:24	i-d1bd21f0	ec2-54-242-9-252.compute-1.amazonaws.com	Kerberos	cassandra_pn_kcs,base,pn_kcs,pn-cassandra
4. running	2014-01-31T16:21	i-eff969c1	ec2-54-205-8-239.compute-1.amazonaws.com	Kerberos	cassandra_pn_kcs,base,pn_kcs,pn-cassandra,pn-cassandra
5. running	2014-01-31T16:21	i-eef969c0	ec2-54-197-186-42.compute-1.amazonaws.com	Kerberos	cassandra_pn_kcs,base,pn_kcs,pn-cassandra
```

Organize each node by AZ.

```
us-east-1e 0. running	2014-01-31T16:27	i-b281df92	ec2-184-72-147-62.compute-1.amazonaws.com
us-east-1e 1. running	2014-01-31T16:27	i-b181df91	ec2-54-226-107-205.compute-1.amazonaws.com
us-east-1d 2. running	2014-01-31T16:24	i-d0bd21f1	ec2-184-72-73-37.compute-1.amazonaws.com
us-east-1d 3. running	2014-01-31T16:24	i-d1bd21f0	ec2-54-242-9-252.compute-1.amazonaws.com
us-east-1a 4. running	2014-01-31T16:21	i-eff969c1	ec2-54-205-8-239.compute-1.amazonaws.com
us-east-1a 5. running	2014-01-31T16:21	i-eef969c0	ec2-54-197-186-42.compute-1.amazonaws.com
```

Create data on the test node based on https://gist.github.com/pcn/8405542.  We'll start in us-east-1a, and move to e, so let's do the build on a system in us-east-1e
```
git clone https://github.com/apache/cassandra.git
cd cassandra
git checkout cassandra-1.2.13
ant
CASSANDRA_INCLUDE=tools/bin/cassandra.in.sh tools/bin/cassandra-stress -d 0.0.0.0 -l 3 -e ONE -o INSERT -n 10000000
```

This takes a good 30 minutes or something like that.  So launch an sedeploy-multi launch, with the min and max set to 0.

```
$ sedeploy-multi -e staging  -r us-east
-1 pn-cassandra.json                                                                                                       
Request:1fb2750b-8aa6-11e3-ac4b-0d61efd5ca20
Request:3e15d819-8aa6-11e3-9e09-9798c95598d1
Request:5d094698-8aa6-11e3-89d0-c3097e898670
```

Look for the name of the current CFN-based ASG name so we can reduce the min size so that we can remove a node.

The naming convention we use is <env>-Cassandra-CassandraGroup[ABCDE]xxx<hex numbers>-<hex numbers> and is un-guessable.  So we need to find that name
```
(awscli)8661 pn@pn-mbp 14:20 ~/venv/awscli $ aws autoscaling describe-auto-scaling-groups  | grep PN-Cassandra-CassandraGro
upA
            "AutoScalingGroupARN": "arn:aws:autoscaling:us-east-1:957700444419:autoScalingGroup:94c24919-88d9-4963-a2cd-5290ebdea371:autoScalingGroupName/PN-Cassandra-CassandraGroupAxxx34346538393536622d386139332d313165332d616437312d623865383536333737356663-ESG37CQOUEOM", 
                    "ResourceId": "PN-Cassandra-CassandraGroupAxxx34346538393536622d386139332d313165332d616437312d623865383536333737356663-ESG37CQOUEOM", 
                    "ResourceId": "PN-Cassandra-CassandraGroupAxxx34346538393536622d386139332d313165332d616437312d623865383536333737356663-ESG37CQOUEOM", 
                    "ResourceId": "PN-Cassandra-CassandraGroupAxxx34346538393536622d386139332d313165332d616437312d623865383536333737356663-ESG37CQOUEOM", 
            "AutoScalingGroupName": "PN-Cassandra-CassandraGroupAxxx34346538393536622d386139332d313165332d616437312d623865383536333737356663-ESG37CQOUEOM", 
```

Now that we've go the name, reduce the min count by one, then remove the node we're killing.
```
$ aws autoscaling update-auto-scaling-group --auto-scaling-group-name PN-Cassandra-CassandraGroupAxxx34346538393536622d386139332d313165332d616437312d623865383536333737356663-ESG37CQOUEOM --min-size 1
$ aws autoscaling terminate-instance-in-auto-scaling-group --instance-id i-eef969c0 --should-decrement-desired-capacity
{
    "Activity": {
        "Description": "Terminating EC2 instance: i-eef969c0", 
        "ActivityId": "93d8d911-f2c6-4f95-9846-c7f188e31520", 
        "Details": "{\"Availability Zone\":\"us-east-1a\"}", 
        "StartTime": "2014-01-31T19:32:57.980Z", 
        "Progress": 0, 
        "Cause": "At 2014-01-31T19:32:57Z instance i-eef969c0 was taken out of service in response to a user request, shrinking the capacity from 2 to 1.", 
        "StatusCode": "InProgress"
    }
}
```

Lower the parameters of the CFN-created autoscale group to min = max = desired so there can't be a scaling event that would bring up a new node.

```
aws autoscaling update-auto-scaling-group --auto-scaling-group-name PN-Cassandra-CassandraGroupAxxx34346538393536622d386139332d313165332d616437312d623865383536333737356663-ESG37CQOUEOM --min-size 1 --max-size 1 --desired-capacity 1
```

2.Work the replacement

Verify that the node is down, then increment the size of the replacement group by one:

```
[staging] (pn-cassandra) pn@ip-10-170-81-79:~ $ nodetool -h0 ring
Address         DC          Rack        Status State   Load            Owns    Token                                       
                                                                               141784319550391026443072753098378663705     
10.9.188.200    us-east     1a          Down   Normal  1.54 GB         16.67%  1808575600                                  
10.118.67.76    us-east     1d          Up     Normal  1.54 GB         16.67%  28356863910078205288614550621122593221      
10.170.81.79    us-east     1e          Up     Normal  1.54 GB         16.67%  56713727820156410577229101240436610842      
10.225.22.208   us-east     1a          Up     Normal  1.53 GB         16.67%  85070591730234615865843651859750628463      
10.98.65.148    us-east     1d          Up     Normal  1.54 GB         16.67%  113427455640312821154458202479064646084     
10.182.188.39   us-east     1e          Up     Normal  1.54 GB         16.67%  141784319550391026443072753098378663705    
```

Now increase the replacement groups count and wait for an instance to come up.
```
$ aws autoscaling update-auto-scaling-group --auto-scaling-group-name pn-cassandra+us-east-1a --min-size 1 --max-size 1 --desired-capacity 1
```

Check to confirm that it's come up:
```
(awscli)8685 pn@pn-mbp 14:43 ~/venv/awscli $ hss s pn-cassandra+us-east-1a
CAN ONLY SSH TO KERBERIZED INSTANCES: 0. running        2014-01-31T19:38        i-9e75e5b0      ec2-50-19-197-7.compute-1.amazonaws.com    RSA-Only        default,cost-center=other
```
or wait...

4 nodes replaced

After 4 nodes are replaced, check the ring, and run a read test with CL=ALL:

```
[staging] (pn-cassandra) pn@ip-10-182-188-39:~/cassandra $ CASSANDRA_INCLUDE=tools/bin/cassandra.in.sh tools/bin/cassandra-
stress -d 0.0.0.0 -l 3 -e ALL -o READ -n 10000000
total,interval_op_rate,interval_key_rate,latency/95th/99.9th,elapsed_time
184,18,18,2120.3,5473.6,7779.7,10
1096,91,91,168.4,3707.7,7779.7,20
1451,35,35,170.1,4523.4,9202.5,30
3010,155,155,96.6,2794.9,9058.8,40
7801,479,479,34.9,1068.4,9058.8,50
13660,585,585,30.0,765.6,9058.8,60
[...]
9960515,2861,2861,15.0,42.0,79.7,4257
9987833,2731,2731,15.0,41.9,77.0,4267
10000000,1216,1216,14.8,41.7,77.0,4272
END
```

Success!

The nodes do need to have unsafe_assassinate_endpoint invoked since this is still 1.0.x.