The AWS Auto Scaling Goup, configured with a customised Cloud-Init file, sends a notification to an SNS Topic, which in turn passes it onto an SQS queue that the Salt Master is subscribed to. A Reactor watches for the auto scaling events and pre-approves the new minion based on its Auto Scaling group name and instance ID.
These changes require the Salt Master be restarted.
/etc/salt/master.d/engines.conf
my_sqs_profile:
region: us-east-1
message_format: json
# Your AWS access key ID and secret key
# for the account that will access the SQS queue
keyid: YOURAWSACCESSID
key: Your4WSS3cr3tK3y1D
engines:
# Add more sqs_events list items to watch more SQS queues
- sqs_events:
queue: my_sqs_queue
profile: my_sqs_profile
tag: salt/engine/sqs/autoscaling/etc/salt/master.d/reactor.conf
reactor:
# Autoscaling reactor based on the SQS engine events
- 'salt/engine/sqs/autoscaling':
- '/srv/salt/reactors/my_ec2_autoscaling.sls'/srv/salt/reactors/my_ec2_autoscaling.sls
#!py
import json
def run():
'''
Run the reactor
'''
ret = {}
sns = json.loads(data['message'])
message = json.loads(sns['Message'])
details = message['Details']
group_name = str(message['AutoScalingGroupName'])
instance_id = str(message['EC2InstanceId'])
if 'launch' in sns['Subject']:
# Fire off an event to wait for the machine
ret = {
'ec2_autoscale_autosign': {
'runner.my_runner.autosign': [{'name': group_name + instance_id}]
}
}
elif 'termination' in sns['Subject']:
ret = {
'ec2_autoscale_termination': {
'wheel.key.delete': [
{'match': group_name + instance_id},
]
}
}
return retThis is the custom runner to pre-approve the new minions. Update your master configuration like so:
module_dirs:
- /srv/salt/extmodsTo allow the use of custom runners (and other custom modules).
/srv/salt/extmods/runners/my_runner.py
'''
A custom runner for Salt Master tasks
'''
import os.path
def autosign(name, output=True):
'''
Create a file in minions_autosign to pre-approve a minion
'''
ret = {}
autosign_key = os.path.join(__opts__['pki_dir'], 'minions_autosign', name)
open(autosign_key, 'a').close()
ret['key'] = autosign_key
return retWhen configuring the Auto Scaling Group, use the cloud-init file at the bottom of the article as the userdata for new instances, replacing the _GRP_ placeholder with the name of your group. New instances will be expected to have their instance ID prefixed with the group name as their hostname and minion ID.
Create the SQS queue that your Salt Master will connect to:
$ aws sqs create-queue --queue-name my_sqs_queue
{
"QueueUrl": "https://queue.amazonaws.com/xxxxxxxxxxxx/my_sqs_queue"
}
We will need to know the ARN for the new queue:
$ aws sqs get-queue-attributes --queue-url https://queue.amazonaws.com/xxxxxxxxxxxx/my_sqs_queue --attribute-names QueueArn
{
"Attributes": {
"QueueArn": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue"
}
}Create an SNS Topic to receive notifications from your Auto Scaling Group:
$ aws sns create-topic --name my_sns_topic
{
"TopicArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic"
}
Using the new SNS Topic ARN, add notification configurations to your Auto Scaling Group:
$ aws autoscaling put-notification-configuration --auto-scaling-group-name my_auto_group --topic-arn arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic --notification-type autoscaling:EC2_INSTANCE_LAUNCH
$ aws autoscaling put-notification-configuration --auto-scaling-group-name my_auto_group --topic-arn arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic --notification-type autoscaling:EC2_INSTANCE_TERMINATE
Now we need to subscribe the new SQS Queue to the SNS Topic:
$ aws sns subscribe --topic-arn arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue
{
"SubscriptionArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
}
Grant the SNS Topic permission to push notifications to the SQS Queue. This requires explicitly setting the policy document so we can set a conditional. You will also need the ARN for the user that the Salt Master will be connecting as. The resulting policy document will be:
{
"Version": "2012-10-17",
"Id": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue/SQSDefaultPolicy",
"Statement": [
{
"Sid": "FromMySNSTopicToMySQSQueue",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "SQS:SendMessage",
"Resource": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue",
"Condition": {
"ArnEquals": {
"aws:SourceArn": "arn:aws:sns:us-east-1:xxxxxxxxxxxx:my_sns_topic"
}
}
},
{
"Sid": "SaltStackReadAccess",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::xxxxxxxxxxxx:user/saltstack.user"
},
"Action": [
"SQS:ReceiveMessage",
"SQS:DeleteMessage",
"SQS:GetQueueUrl"
],
"Resource": "arn:aws:sqs:us-east-1:xxxxxxxxxxxx:my_sqs_queue"
}
]
}
The permissions on the SQS have been updated. The Saltstack SQS engine requires permission to both receive and delete messages.