# A SaltStack AWS Auto Scaling Solution

## Overview

The AWS Auto Scaling Goup, configured with a customised Cloud-Init file, sends a notification to an SNS Topic,
which in turn passes it onto an SQS queue that the Salt Master is subscribed to. A Reactor watches for the auto
scaling events and pre-approves the new minion based on its Auto Scaling group name and instance ID.

## Salt Master Configuration

These changes require the Salt Master be restarted.

### SQS Engine
`/etc/salt/master.d/engines.conf`
```yaml
my_sqs_profile:
  region: us-east-1
  message_format: json
  # Your AWS access key ID and secret key
  # for the account that will access the SQS queue
  keyid: YOURAWSACCESSID
  key: Your4WSS3cr3tK3y1D

engines:
# Add more sqs_events list items to watch more SQS queues
- sqs_events:
    queue: my_sqs_queue
    profile: my_sqs_profile
    tag: salt/engine/sqs/autoscaling
```

### Reactor
`/etc/salt/master.d/reactor.conf`
```yaml
reactor:
  # Autoscaling reactor based on the SQS engine events
  - 'salt/engine/sqs/autoscaling':
    - '/srv/salt/reactors/my_ec2_autoscaling.sls'
```

## Reactor
`/srv/salt/reactors/my_ec2_autoscaling.sls`
```python
#!py
import json

def run():
	'''
	Run the reactor
	'''
	ret = {}

	sns = json.loads(data['message'])
	message = json.loads(sns['Message'])
	details = message['Details']

	group_name = str(message['AutoScalingGroupName'])
	instance_id = str(message['EC2InstanceId'])

	if 'launch' in sns['Subject']:
		# Fire off an event to wait for the machine
		ret = {
			'ec2_autoscale_autosign': {
				'runner.my_runner.autosign': [{'name': group_name + instance_id}]
			}
		}
	elif 'termination' in sns['Subject']:
		ret = {
			'ec2_autoscale_termination': {
				'wheel.key.delete': [
					{'match': group_name + instance_id},
				]
			}
		}

	return ret
```

## Custom Runner
This is the custom runner to pre-approve the new minions. Update your master configuration like so:
```yaml
module_dirs:
  - /srv/salt/extmods
```
To allow the use of custom runners (and other custom modules).

`/srv/salt/extmods/runners/my_runner.py`
```python
'''
A custom runner for Salt Master tasks
'''
import os.path

def autosign(name, output=True):
    '''
    Create a file in minions_autosign to pre-approve a minion
    '''
    ret = {}
    autosign_key = os.path.join(__opts__['pki_dir'], 'minions_autosign', name)
    open(autosign_key, 'a').close()

    ret['key'] = autosign_key

    return ret
```

## AWS Configuration


### Auto Scaling Group
When configuring the Auto Scaling Group, use the [cloud-init file at the bottom of the article](#file-cloud-init-yaml) as the userdata for new instances, replacing the `_GRP_` placeholder with the name of your group. New instances will be expected to have their instance ID prefixed with the group name as their hostname and minion ID.

### SQS Queue
Create the SQS queue that your Salt Master will connect to:
```
$ aws sqs create-queue --queue-name my_sqs_queue
{
    "QueueUrl": "https://queue.amazonaws.com/xxxxxxxxxxxx/my_sqs_queue"
}
```

We will need to know the ARN for the new queue:
```bash
$ aws sqs get-queue-attributes --queue-url https://queue.amazonaws.com/xxxxxxxxxxxx/my_sqs_queue --attribute-names QueueArn
{
    "Attributes": {
        "QueueArn": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue"
    }
}
```


### SNS Topic
Create an SNS Topic to receive notifications from your Auto Scaling Group:
```
$ aws sns create-topic --name my_sns_topic
{
    "TopicArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic"
}
```


### Auto Scaling Notifications
Using the new SNS Topic ARN, add notification configurations to your Auto Scaling Group:
```
$ aws autoscaling put-notification-configuration --auto-scaling-group-name my_auto_group --topic-arn arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic --notification-type autoscaling:EC2_INSTANCE_LAUNCH
$ aws autoscaling put-notification-configuration --auto-scaling-group-name my_auto_group --topic-arn arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic --notification-type autoscaling:EC2_INSTANCE_TERMINATE
```


### Subscription
Now we need to subscribe the new SQS Queue to the SNS Topic:
```
$ aws sns subscribe --topic-arn arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue
{
    "SubscriptionArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
```


### Permissions
Grant the SNS Topic permission to push notifications to the SQS Queue. This requires explicitly setting the policy document so we can set a conditional. You will also need the ARN for the user that the Salt Master will be connecting as. The resulting policy document will be:
```json
{
  "Version": "2012-10-17",
  "Id": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue/SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "FromMySNSTopicToMySQSQueue",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic"
        }
      }
    },
    {
      "Sid": "SaltStackReadAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxxx:user/saltstack.user"
      },
      "Action": "SQS:ReceiveMessage",
      "Resource": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue"
    }
  ]
}
```