Skip to content

Instantly share code, notes, and snippets.

@gordonclarkok
Created June 21, 2018 09:12
Show Gist options
  • Save gordonclarkok/3c30963ad07f5f38282d76e2371a23b1 to your computer and use it in GitHub Desktop.
Save gordonclarkok/3c30963ad07f5f38282d76e2371a23b1 to your computer and use it in GitHub Desktop.
My example AWS EMR cluster in cloudformation - security groups in separate file
AWSTemplateFormatVersion: 2010-09-09
Description: EMR Cluster with Hadoop, Hbase, Spark, Ganglia and Zookeeper
Parameters:
EMRClusterName:
Description: Name of the cluster
Type: String
Default: emrcluster
KeyName:
Description: Must be an existing Keyname
Type: String
Default: mykey
MasterInstanceType:
Description: Instance type to be used for the master instance.
Type: String
Default: m4.large
CoreInstanceType:
Description: Instance type to be used for core instances.
Type: String
Default: m4.large
NumberOfCoreInstances:
Description: Must be a valid number
Type: Number
Default: 2
LogUri:
Description: Must be a valid S3 URL
Default: 's3://my-emr-logbucket/'
Type: String
S3DataUri:
Description: 'Must be a valid S3 bucket URL '
Default: 's3://my-emr/'
Type: String
ReleaseLabel:
Description: Must be a valid EMR release version
Default: emr-5.11.0
Type: String
Resources:
EMRCluster:
DependsOn:
- EMRClusterServiceRole
- EMRClusterinstanceProfileRole
- EMRClusterinstanceProfile
Type: 'AWS::EMR::Cluster'
Properties:
Applications:
- Name: Spark
- Name: Hbase
- Name: Hadoop
- Name: Ganglia
- Name: Zookeeper
Configurations:
- Classification: hbase-site
ConfigurationProperties:
hbase.rootdir: !Ref S3DataUri
- Classification: hbase
ConfigurationProperties:
hbase.emr.storageMode: s3
Instances:
Ec2KeyName: !Ref KeyName
Ec2SubnetId: !ImportValue MySubnetPrivateA
EmrManagedMasterSecurityGroup: !ImportValue EmrMasterSgId
AdditionalMasterSecurityGroups:
- !ImportValue EmrMasterAdditionalSgId
EmrManagedSlaveSecurityGroup: !ImportValue EmrSlaveSgId
AdditionalSlaveSecurityGroups:
- !ImportValue EmrSlaveAdditionalSgId
ServiceAccessSecurityGroup: !ImportValue EmrServiceSgId
MasterInstanceGroup:
InstanceCount: 1
InstanceType: !Ref MasterInstanceType
Market: ON_DEMAND
Name: Master
CoreInstanceGroup:
InstanceCount: !Ref NumberOfCoreInstances
InstanceType: !Ref CoreInstanceType
Market: ON_DEMAND
Name: Core
TerminationProtected: false
VisibleToAllUsers: true
JobFlowRole: !Ref EMRClusterinstanceProfile
ReleaseLabel: !Ref ReleaseLabel
LogUri: !Ref LogUri
Name: !Ref EMRClusterName
AutoScalingRole: EMR_AutoScaling_DefaultRole
ServiceRole: !Ref EMRClusterServiceRole
Tags:
-
Key: "cluster_name"
Value: "master.emr.my.com"
EMRClusterServiceRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- elasticmapreduce.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole'
Path: /
EMRClusterinstanceProfileRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- 'sts:AssumeRole'
ManagedPolicyArns:
- 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role'
Path: /
EMRClusterinstanceProfile:
Type: 'AWS::IAM::InstanceProfile'
Properties:
Path: /
Roles:
- !Ref EMRClusterinstanceProfileRole
Outputs: {}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment