Skip to content

Instantly share code, notes, and snippets.

@jheller

jheller/Ansible CloudFormation Macros.md

Last active January 4, 2023 19:37
Show Gist options
  • Save jheller/c4fa0075e4eccf094769 to your computer and use it in GitHub Desktop.
Save jheller/c4fa0075e4eccf094769 to your computer and use it in GitHub Desktop.
Download ZIP
Using Ansible/Jinja macros to generate CloudFormation templates
Raw
Ansible CloudFormation Macros.md

Generating CloudFormation Templates from YAML Dictionaries

The security group and Network ACLs parts of CloudFormation templates can be difficult to read. This template containing Jinja macros converts easier-to-read YAML dictionaries of security group and NACL rules into JSON.

Here ais a generic macro template and some example files showing how to use it.

  • cloudformation.macros.j2 - the macros
  • cf_vars.yml - YAML dictionaly of security groups and NACLs, including some complex rules for both
  • test.template.j2 - a simple CloudFormation template that uses the macro file
  • cf_test.yml - a simple Ansible playbook that generates CloudFormation JSON from thr previous file.

Run it like this to generate a CloudFormation template in test.template:-

ansible-playbook cf_test.yml

One day soon, when time permits I will expand the macros to include other AWS networking resources such as subnets and route tables.

Ansible 2.0

This came out recently and the new CloudFormation module accepts templates defined in YAML. It does a simple YAML->JSON conversion before uploading the result. I still think this is a little more readable when you are managing complex security group rules.

Raw
cf_test.yml
---
- name: Provision Stack
hosts: local # use ansible installed locally
gather_facts: False
vars_files:
- cf_vars.yml
tasks:
- name: Run template
template:
src: "test.template.j2"
dest: "./test.template"
Raw
cf_vars.yml
---
#Group
vpc_id: vpc-12345678
peer_cidr: "10.1.2.3/24"
SubnetPubACIDR: "10.1.0.0/24"
SubnetPubBCIDR: "10.1.1.0/24"
SubnetAppACIDR: "10.1.2.0/24"
SubnetAppBCIDR: "10.1.3.0/24"
SubnetDbACIDR: "10.1.4.0/24"
SubnetDbBCIDR: "10.1.5.0/24"
SubnetToolACIDR: "10.1.6.0/24"
SubnetToolBCIDR: "10.1.7.0/24"
security_groups:
CommonMgmtSg:
tags:
barry: robert
kevin: rudd
ingress:
HISSharedVPCHTTP: { Proto: tcp, From: 80, To: 80, Cidr: "{{ peer_cidr }}" }
HISSharedVPCHTTPS: { Proto: tcp, From: 443, To: 443, Cidr: "{{ peer_cidr }}" }
HISSharedVPCSSH: { Proto: tcp, From: 22, To: 22, Cidr: "{{ peer_cidr }}" }
egress:
HISSharedVPCProxy: { Proto: tcp, From: 3128, To: 3128, Cidr: "{{ peer_cidr }}" }
WebElbSg:
ingress:
ExternalHTTP: { Proto: tcp, From: 80, To: 80, Cidr: "0.0.0.0/0" }
ExternalHTTPS: { Proto: tcp, From: 443, To: 443, Cidr: "0.0.0.0/0" }
egress:
WebSgHTTP: { Proto: tcp, From: 80, To: 80, Group: WebSg }
WebSgHTTPS: { Proto: tcp, From: 443, To: 443, Group: WebSg }
WebSg:
ingress:
WebElbSgHTTP: { Proto: tcp, From: 80, To: 80, Group: WebElbSg }
WebElbSgHTTPS: { Proto: tcp, From: 443, To: 443, Group: WebElbSg }
AppSgHTTP: { Proto: tcp, From: 80, To: 80, Group: AppSg }
AppSgHTTPS: { Proto: tcp, From: 443, To: 443, Group: AppSg }
AppSgTCP8008: { Proto: tcp, From: 8008, To: 8008, Group: AppSg }
egress:
WebSgHTTP: { Proto: tcp, From: 80, To: 80, Group: AppSg }
WebSgHTTPS: { Proto: tcp, From: 443, To: 443, Group: AppSg }
AppSgWASApp: { Proto: tcp, From: 17000, To: 19000, Group: AppSg }
AppElbSg:
ingress:
ExternalTCP9043: { Proto: tcp, From: 9043, To: 9043, Cidr: "0.0.0.0/0" }
egress:
AppSgTCP9043: { Proto: tcp, From: 9043, To: 9043, Group: AppSg }
AppSg:
ingress:
AppElbSgTCP9043: { Proto: tcp, From: 9043, To: 9043, Group: AppElbSg }
AppSgHTTP: { Proto: tcp, From: 80, To: 80, Group: AppSg }
AppSgHTTPS: { Proto: tcp, From: 443, To: 443, Group: AppSg }
AppSgTCP8008: { Proto: tcp, From: 17000, To: 17000, Group: AppSg }
WebSgWASApp: { Proto: tcp, From: 17000, To: 19000, Group: WebSg }
WebSgHTTP: { Proto: tcp, From: 80, To: 80, Group: WebSg }
WebSgHTTPS: { Proto: tcp, From: 443, To: 443, Group: WebSg }
egress:
AppSgTCP9043: { Proto: tcp, From: 9043, To: 9043, Group: AppSg }
AppSgTCP8008: { Proto: tcp, From: 8008, To: 8008, Group: WebSg }
DbSg:
existing: true
ingress:
AppSgDB: { Proto: tcp, From: 1026, To: 65535, Group: AppElbSg }
nacl_rules:
PubAcl:
ingress:
PubSunetA: { Rule: 600, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetPubACIDR }}" }
PubSunetB: { Rule: 601, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetPubBCIDR }}" }
WebSphereAdmin: { Rule: 1903, Action: allow, Proto: "{{ tcp }}", Cidr: "0.0.0.0/0", From: 9043, To: 9043 }
ExternalHTTP: { Rule: 1900, Action: allow, Proto: "{{ tcp }}", Cidr: "0.0.0.0/0", From: 80, To: 80 }
ExternalHTTPS: { Rule: 1901, Action: allow, Proto: "{{ tcp }}", Cidr: "0.0.0.0/0", From: 443, To: 443 }
ExternalSSH: { Rule: 1902, Action: allow, Proto: "{{ tcp }}", Cidr: "0.0.0.0/0", From: 22, To: 22 }
AppSubnetAEph: { Rule: 2000, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetAppACIDR }}", From: 1024, To: 65535 }
AppSubnetBEph: { Rule: 2001, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetAppBCIDR }}", From: 1024, To: 65535 }
egress:
PubSunetA: { Rule: 600, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetPubACIDR }}" }
PubSunetB: { Rule: 601, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetPubBCIDR }}" }
ExternalHTTP: { Rule: 1900, Action: allow, Proto: "{{ tcp }}", Cidr: "0.0.0.0/0", From: 80, To: 80 }
ExternalHTTPS: { Rule: 1901, Action: allow, Proto: "{{ tcp }}", Cidr: "0.0.0.0/0", From: 443, To: 443 }
ExternalSSH: { Rule: 1902, Action: allow, Proto: "{{ tcp }}", Cidr: "0.0.0.0/0", From: 22, To: 22 }
AppSubnetAEph: { Rule: 2000, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetAppACIDR }}", From: 1024, To: 65535 }
AppSubnetBEph: { Rule: 2001, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetAppBCIDR }}", From: 1024, To: 65535 }
ExternalEph: { Rule: 2002, Action: allow, Proto: "{{ tcp }}", Cidr: "0.0.0.0/0", From: 1024, To: 65535 }
AppAcl:
ingress:
HTTPPubA: { Rule: 400, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetPubACIDR }}", From: 80, To: 80 }
HTTPPubB: { Rule: 401, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetPubBCIDR }}", From: 80, To: 80 }
HTTPSPubA: { Rule: 402, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetPubACIDR }}", From: 443, To: 443 }
HTTPSPubB: { Rule: 403, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetPubBCIDR }}", From: 443, To: 443 }
WASPubA: { Rule: 404, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetPubACIDR }}", From: 9043, To: 9043 }
WASPubB: { Rule: 405, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetPubBCIDR }}", From: 9043, To: 9043 }
AppSubnetA: { Rule: 600, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppACIDR }}" }
AppSubnetB: { Rule: 601, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppBCIDR }}" }
ToolSubnetA: { Rule: 602, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolACIDR }}" }
ToolSubnetB: { Rule: 603, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolBCIDR }}" }
DbSubnetA: { Rule: 604, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbACIDR }}" }
DbSubnetB: { Rule: 605, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbBCIDR }}" }
MgmtVPC: { Rule: 1200, Action: allow, Proto: "{{ all }}", Cidr: "{{ peer_cidr }}" }
NABHTTP: { Rule: 1401, Action: allow, Proto: "{{ tcp }}", Cidr: "10.96.0.0/12", From: 80, To: 80 }
NABHTTPS: { Rule: 1402, Action: allow, Proto: "{{ tcp }}", Cidr: "10.96.0.0/12", From: 443, To: 443 }
NABWasAdminFrom: { Rule: 1403, Action: allow, Proto: "{{ tcp }}", Cidr: "10.96.0.0/12", From: 9043, To: 9043 }
egress:
PubAEph: { Rule: 500, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetPubACIDR }}", From: 1024, To: 65535 }
PubBEph: { Rule: 501, Action: allow, Proto: "{{ tcp }}", Cidr: "{{ SubnetPubBCIDR }}", From: 1024, To: 65535 }
AppSubnetA: { Rule: 600, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppACIDR }}" }
AppSubnetB: { Rule: 601, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppBCIDR }}" }
ToolSubnetA: { Rule: 602, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolACIDR }}" }
ToolSubnetB: { Rule: 603, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolBCIDR }}" }
DbSubnetA: { Rule: 604, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbACIDR }}" }
DbSubnetB: { Rule: 605, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbBCIDR }}" }
MgmtVPC: { Rule: 1200, Action: allow, Proto: "{{ all }}", Cidr: "{{ peer_cidr }}" }
NABHTTP: { Rule: 1401, Action: allow, Proto: "{{ tcp }}", Cidr: "10.96.0.0/12", From: 80, To: 80 }
NABHTTPS: { Rule: 1402, Action: allow, Proto: "{{ tcp }}", Cidr: "10.96.0.0/12", From: 443, To: 443 }
NABWasAdminFrom: { Rule: 1403, Action: allow, Proto: "{{ tcp }}", Cidr: "10.96.0.0/12", From: 9043, To: 9043 }
NABEphemeral: { Rule: 2000, Action: allow, Proto: "{{ tcp }}", Cidr: "10.96.0.0/12", From: 1024, To: 65535 }
DbAcl:
ingress:
AppSubnetA: { Rule: 600, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppACIDR }}" }
AppSubnetB: { Rule: 601, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppBCIDR }}" }
ToolSubnetA: { Rule: 602, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolACIDR }}" }
ToolSubnetB: { Rule: 603, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolBCIDR }}" }
DbSubnetA: { Rule: 604, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbACIDR }}" }
DbSubnetB: { Rule: 605, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbBCIDR }}" }
MgmtVPC: { Rule: 1200, Action: allow, Proto: "{{ all }}", Cidr: "{{ peer_cidr }}" }
egress:
AppSubnetA: { Rule: 600, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppACIDR }}" }
AppSubnetB: { Rule: 601, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppBCIDR }}" }
ToolSubnetA: { Rule: 602, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolACIDR }}" }
ToolSubnetB: { Rule: 603, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolBCIDR }}" }
DbSubnetA: { Rule: 604, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbACIDR }}" }
DbSubnetB: { Rule: 605, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbBCIDR }}" }
MgmtVPC: { Rule: 1200, Action: allow, Proto: "{{ all }}", Cidr: "{{ peer_cidr }}" }
ToolAcl:
ingress:
AppSubnetA: { Rule: 600, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppACIDR }}" }
AppSubnetB: { Rule: 601, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppBCIDR }}" }
ToolSubnetA: { Rule: 602, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolACIDR }}" }
ToolSubnetB: { Rule: 603, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolBCIDR }}" }
DbSubnetA: { Rule: 604, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbACIDR }}" }
DbSubnetB: { Rule: 605, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbBCIDR }}" }
MgmtVPC: { Rule: 1200, Action: allow, Proto: "{{ all }}", Cidr: "{{ peer_cidr }}" }
egress:
AppSubnetA: { Rule: 600, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppACIDR }}" }
AppSubnetB: { Rule: 601, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetAppBCIDR }}" }
ToolSubnetA: { Rule: 602, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolACIDR }}" }
ToolSubnetB: { Rule: 603, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetToolBCIDR }}" }
DbSubnetA: { Rule: 604, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbACIDR }}" }
DbSubnetB: { Rule: 605, Action: allow, Proto: "{{ all }}", Cidr: "{{ SubnetDbBCIDR }}" }
MgmtVPC: { Rule: 1200, Action: allow, Proto: "{{ all }}", Cidr: "{{ peer_cidr }}" }
Raw
cloudformation.macros.j2
{%- macro securitygroups(security_groups) -%}
{% for sg in security_groups %}
{% if security_groups[sg].existing is not defined %}
"{{ sg }}": {
"Type": "AWS::EC2::SecurityGroup",
"Properties": {
"GroupDescription": "WAS Application Server(s) Security Group",
"VpcId": "{{ vpc_id }}",
"Tags": [
{
"Key": "Name",
"Value": { "Fn::Join": [ "-", [ { "Ref": "AWS::StackName" }, "{{ sg }}" ]]}
}
{%- if security_groups[sg].tags is defined -%}
{%- for tag in security_groups[sg].tags -%}
,
{
"Key": "{{ tag }}",
"Value": "{{ security_groups[sg].tags[tag] }}"
}{{ "\n" if loop.last else "" }}
{%- endfor -%}
{%- else -%}{{ "\n" }}
{%- endif %}
]
}
},
{% endif %}
{% endfor %}
{% for sg in security_groups %}
{% set sg_loop = loop %}
{% if security_groups[sg].ingress is defined %}
{% for rule in security_groups[sg].ingress %}
"{{ sg }}Ingress{{ rule }}": {
"Type": "AWS::EC2::SecurityGroupIngress",
"Properties": {
"GroupId": { "Ref": "{{ sg }}" },
"IpProtocol": "{{ security_groups[sg].ingress[rule].Proto }}",
"FromPort": "{{ security_groups[sg].ingress[rule].From }}",
"ToPort": "{{ security_groups[sg].ingress[rule].To }}",
{% if security_groups[sg].ingress[rule].Group is defined %}
"SourceSecurityGroupId": { "Ref": "{{ security_groups[sg].ingress[rule].Group }}" }
{% elif security_groups[sg].ingress[rule].Cidr is defined %}
"CidrIp": "{{ security_groups[sg].ingress[rule].Cidr }}"
{% endif %}
}
},
{% endfor %}
{% endif %}
{% endfor %}
{% for sg in security_groups %}
{% set sg_loop = loop %}
{% if security_groups[sg].egress is defined %}
{% for rule in security_groups[sg].egress %}
"{{ sg }}Egress{{ rule }}": {
"Type": "AWS::EC2::SecurityGroupEgress",
"Properties": {
"GroupId": { "Ref": "{{ sg }}" },
"IpProtocol": "{{ security_groups[sg].egress[rule].Proto }}",
"FromPort": "{{ security_groups[sg].egress[rule].From }}",
"ToPort": "{{ security_groups[sg].egress[rule].To }}",
{% if security_groups[sg].egress[rule].Group is defined %}
"DestinationSecurityGroupId": { "Ref": "{{ security_groups[sg].egress[rule].Group }}" }
{% elif security_groups[sg].egress[rule].Cidr is defined %}
"CidrIp": "{{ security_groups[sg].egress[rule].Cidr }}"
{% endif %}
}
}{{ "" if loop.last and sg_loop.last else ",\n" }}
{%- endfor -%}
{%- endif -%}
{%- endfor -%}
{%- endmacro -%}
{% macro nacls(nacl_rules) -%}
{% for acl in nacl_rules %}
{% if nacl_rules[acl].existing is not defined %}
"{{ acl }}": {
"Type": "AWS::EC2::NetworkAcl",
"Properties": {
"VpcId": "{{ vpc_id }}",
"Tags": [
{
"Key": "Name",
"Value": { "Fn::Join": [ "-", [ { "Ref": "AWS::StackName" }, "{{ acl }}" ] ] }
}
{%- if nacl_rules[acl].tags is defined -%}
{%- for tag in nacl_rules[acl].tags -%}
,
{
"Key": "{{ tag }}",
"Value": "{{ nacl_rules[acl].tags[tag] }}"
}{{ "\n" if loop.last else "" }}
{%- endfor -%}
{%- else -%}{{ "\n" }}
{%- endif %}
]
}
},
{% endif %}
{% endfor %}
{% for acl in nacl_rules %}
{% if nacl_rules[acl].ingress is defined %}
{% for rule in nacl_rules[acl].ingress %}
"{{ acl }}Ingress{{ rule }}": {
"Type": "AWS::EC2::NetworkAclEntry",
"Properties": {
"NetworkAclId": {
"Ref": "{{ acl }}"
},
"RuleNumber": "{{ nacl_rules[acl].ingress[rule].Rule }}",
"Protocol": "{{ nacl_rules[acl].ingress[rule].Proto }}",
"RuleAction": "{{ nacl_rules[acl].ingress[rule].Action }}",
"Egress": "false",
{% if nacl_rules[acl].ingress[rule].From is defined %}
"PortRange": {
"From": "{{ nacl_rules[acl].ingress[rule].From }}",
"To": "{{ nacl_rules[acl].ingress[rule].To }}"
},
{% endif %}
"CidrBlock": "{{ nacl_rules[acl].ingress[rule].Cidr }}"
}
},
{% endfor %}
{% endif %}
{% endfor %}
{% for acl in nacl_rules %}
{% set acl_loop = loop %}
{% if nacl_rules[acl].egress is defined %}
{% for rule in nacl_rules[acl].egress %}
"{{ acl }}Egress{{ rule }}": {
"Type": "AWS::EC2::NetworkAclEntry",
"Properties": {
"NetworkAclId": {
"Ref": "{{ acl }}"
},
"RuleNumber": "{{ nacl_rules[acl].egress[rule].Rule }}",
"Protocol": "{{ nacl_rules[acl].egress[rule].Proto }}",
"RuleAction": "{{ nacl_rules[acl].egress[rule].Action }}",
"Egress": "true",
{% if nacl_rules[acl].egress[rule].From is defined %}
"PortRange": {
"From": "{{ nacl_rules[acl].egress[rule].From }}",
"To": "{{ nacl_rules[acl].egress[rule].To }}"
},
{% endif %}
"CidrBlock": "{{ nacl_rules[acl].egress[rule].Cidr }}"
}
}{{ "" if loop.last and acl_loop.last else ",\n" }}
{%- endfor -%}
{%- endif -%}
{%- endfor -%}
{%- endmacro %}
Raw
test.template.j2
{% import 'cloudformation.macros.j2' as cf with context %}
{
"Description": "Create complete Web and App environment",
"Resources": {
{{ cf.securitygroups(security_groups) }},
{{ cf.nacls(nacl_rules) }}
}
}
@teddevaal
Copy link

Please define all and tcp inside cf_vars.yml

all: "-1"
tcp: "6"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment