AWS ElasticSearch Domain with VPC and Bastion Cloudformation Template

elasticsearch-service-template.yaml

# Copyright 2020 Daniel Cortez Stevenson
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
---
AWSTemplateFormatVersion: 2010-09-09

Description: >-
  Provisions and deploys AWS Elasticsearch Service (ES) over multiple
  availability zones in a new AWS Virtual Private Cloud (VPC), along with a
  bastion server from which Elasticsearch can be accessed.

Parameters:

  ElasticsearchName:
    Description: >-
      The name of the AWS Elasticsearch Service deployment.
    Type: String
    ConstraintDescription: >-
      Must be a valid AWS ES domain name prefix. The name must start with a
      lowercase letter and must be between 3 and 28 characters. Valid characters
      are a-z (lowercase only), 0-9, and - (hyphen).
    AllowedPattern: '[a-z][a-z0-9\\-]+'

  ElasticsearchVersion:
    Default: 7.4
    Type: String
    ConstraintDescription: >-
      Must be an allowed AWS ES version (Major.Minor)
    AllowedValues:
      - 7.4
      - 7.1
      - 6.8
      - 6.7
      - 6.5
      - 6.4
      - 6.3
      - 6.2
      - 6.0
      - 5.6
      - 5.5
      - 5.3
      - 5.1
      - 2.3
      - 1.5

  ElasticsearchMasterInstanceType:
    Description: >-
      Instance type for master nodes.
    Type: String
    Default: t2.small.elasticsearch
    AllowedValues:
      - r5.large.elasticsearch
      - r5.xlarge.elasticsearch
      - r5.2xlarge.elasticsearch
      - r5.4xlarge.elasticsearch
      - r5.12xlarge.elasticsearch
      - t2.small.elasticsearch
      - t2.medium.elasticsearch
      - c4.large.elasticsearch
      - c4.xlarge.elasticsearch
      - c4.2xlarge.elasticsearch
      - c4.4xlarge.elasticsearch
      - c4.8xlarge.elasticsearch
      - i2.xlarge.elasticsearch
      - i2.2xlarge.elasticsearch
      - m4.large.elasticsearch
      - m4.xlarge.elasticsearch
      - m4.2xlarge.elasticsearch
      - m4.4xlarge.elasticsearch
      - m4.10xlarge.elasticsearch
      - r4.large.elasticsearch
      - r4.xlarge.elasticsearch
      - r4.2xlarge.elasticsearch
      - r4.4xlarge.elasticsearch
      - r4.8xlarge.elasticsearch
      - r4.16xlarge.elasticsearch
      - m3.medium.elasticsearch
      - m3.large.elasticsearch
      - m3.xlarge.elasticsearch
      - m3.2xlarge.elasticsearch
      - r3.large.elasticsearch
      - r3.xlarge.elasticsearch
      - r3.2xlarge.elasticsearch
      - r3.4xlarge.elasticsearch
      - r3.8xlarge.elasticsearch

  ElasticsearchDataInstanceType:
    Description: >-
      Instance type for data nodes.
    Type: String
    Default: r5.2xlarge.elasticsearch
    AllowedValues:
      - r5.large.elasticsearch
      - r5.xlarge.elasticsearch
      - r5.2xlarge.elasticsearch
      - r5.4xlarge.elasticsearch
      - r5.12xlarge.elasticsearch
      - t2.small.elasticsearch
      - t2.medium.elasticsearch
      - c4.large.elasticsearch
      - c4.xlarge.elasticsearch
      - c4.2xlarge.elasticsearch
      - c4.4xlarge.elasticsearch
      - c4.8xlarge.elasticsearch
      - i2.xlarge.elasticsearch
      - i2.2xlarge.elasticsearch
      - m4.large.elasticsearch
      - m4.xlarge.elasticsearch
      - m4.2xlarge.elasticsearch
      - m4.4xlarge.elasticsearch
      - m4.10xlarge.elasticsearch
      - r4.large.elasticsearch
      - r4.xlarge.elasticsearch
      - r4.2xlarge.elasticsearch
      - r4.4xlarge.elasticsearch
      - r4.8xlarge.elasticsearch
      - r4.16xlarge.elasticsearch
      - m3.medium.elasticsearch
      - m3.large.elasticsearch
      - m3.xlarge.elasticsearch
      - m3.2xlarge.elasticsearch
      - r3.large.elasticsearch
      - r3.xlarge.elasticsearch
      - r3.2xlarge.elasticsearch
      - r3.4xlarge.elasticsearch
      - r3.8xlarge.elasticsearch


  NumberOfMasterNodes:
    Description: >-
      How many dedicated master nodes you want to have. 3 is recommended.
    Type: Number
    Default: 3
    AllowedValues:
      - 3
      - 5

  NumberOfDataNodes:
    Description: >-
      How many data nodes you want to have. Multiples of your number of
      availability zones (2) is recommended.
    Type: Number
    Default: 2
    MinValue: 1
    MaxValue: 20

  BastionKeyName:
    Description: >-
      Name of the EC2 KeyPair, which enables SSH access to the bastion server.
    Type: AWS::EC2::KeyPair::KeyName
    ConstraintDescription: >-
      Must be the name of an existing EC2 KeyPair.

  AccessLocation:
    Description: >-
      CIDR range from which SSH access to the bastion server is allowed. Default
      is anywhere.
    Type: String
    Default: '0.0.0.0/0'
    ConstraintDescription: >-
      Must be a valid CIDR range.
    MinLength: 9
    MaxLength: 18
    AllowedPattern: '(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})/(\d{1,2})'

  AvailabilityZone1:
    Type: String
    ConstraintDescription: >-
      Must be a valid AWS Availability Zone in eu-central-1 or us-east-1.
    Default: eu-central-1a
    AllowedValues:
      - eu-central-1a
      - eu-central-1b
      - eu-central-1c
      - us-east-1a
      - us-east-1b
      - us-east-1c
      - us-east-1d
      - us-east-1e
      - us-east-1f

  AvailabilityZone2:
    Type: String
    ConstraintDescription: >-
      Must be a valid AWS Availability Zone in eu-central-1 or us-east-1.
    Default: eu-central-1b
    AllowedValues:
      - eu-central-1a
      - eu-central-1b
      - eu-central-1c
      - us-east-1a
      - us-east-1b
      - us-east-1c
      - us-east-1d
      - us-east-1e
      - us-east-1f

Mappings:
  CidrConfig:
    VPC:
      CIDR: 10.192.0.0/16
    Public1:
      CIDR: 10.192.10.0/24
    Public2:
      CIDR: 10.192.11.0/24

  AWSRegionArch2AMI:
    eu-central-1:
      HVM64: ami-054e21e355db24124
    us-east-1:
      HVM64: ami-0fba9b33b5304d8b4

Resources:

  VPC:
    Type: AWS::EC2::VPC
    Properties:
      EnableDnsSupport: true
      EnableDnsHostnames: true
      InstanceTenancy: default
      CidrBlock: !FindInMap
        - CidrConfig
        - VPC
        - CIDR

  InternetGateway:
    Type: AWS::EC2::InternetGateway

  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      VpcId: !Ref VPC
      InternetGatewayId: !Ref InternetGateway

  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Ref AvailabilityZone1
      CidrBlock: !FindInMap
        - CidrConfig
        - Public1
        - CIDR
      MapPublicIpOnLaunch: true

  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Ref AvailabilityZone2
      CidrBlock: !FindInMap
        - CidrConfig
        - Public2
        - CIDR
      MapPublicIpOnLaunch: true

  PublicRouteTable1:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC

  PublicRouteTable2:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC

  PublicSubnetRouteTableAssociation1:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet1
      RouteTableId: !Ref PublicRouteTable1

  PublicSubnetRouteTableAssociation2:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PublicSubnet2
      RouteTableId: !Ref PublicRouteTable2

  PublicRoute1:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable1
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  PublicRoute2:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable2
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  BastionElasticIp:
    Type: AWS::EC2::EIP

  BastionElasticIPAssociation:
    Type: AWS::EC2::EIPAssociation
    Properties:
      InstanceId: !Ref BastionInstance
      EIP: !Ref BastionElasticIp

  BastionInstance:
    Type: AWS::EC2::Instance
    DependsOn: InternetGatewayAttachment
    Properties:
      InstanceType: t3.nano
      KeyName: !Ref BastionKeyName
      SubnetId: !Ref PublicSubnet1
      SecurityGroupIds:
        - !Ref BastionSecurityGroup
      ImageId: !FindInMap
        - AWSRegionArch2AMI
        - !Ref AWS::Region
        - HVM64
      Monitoring: true
      BlockDeviceMappings:
        - DeviceName: /dev/sda1
          Ebs:
            VolumeSize: 8

  BastionSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: >-
        Allows all ingress from AccessLocation.
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: -1
          ToPort: -1
          CidrIp: !Ref AccessLocation

  ReceiveFromBastionSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: >-
        Allows all ingress from BastionSecurityGroup
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: -1
          ToPort: -1
          SourceSecurityGroupId: !Ref BastionSecurityGroup

  ElasticsearchDomain:
    Type: AWS::Elasticsearch::Domain
    Properties:
      ElasticsearchVersion: !Ref ElasticsearchVersion
      DomainName: !Ref ElasticsearchName
      EBSOptions:
        EBSEnabled: true
        Iops: 2000
        VolumeSize: 196
        VolumeType: io1
      ElasticsearchClusterConfig:
        DedicatedMasterEnabled: true
        DedicatedMasterType: !Ref ElasticsearchMasterInstanceType
        DedicatedMasterCount: 3
        InstanceCount: !Ref NumberOfDataNodes
        InstanceType: !Ref ElasticsearchDataInstanceType
        ZoneAwarenessEnabled: true
      AdvancedOptions:
        rest.action.multi.allow_explicit_index: 'true'
        indices.fielddata.cache.size: '40'
        indices.query.bool.max_clause_count: '1024'
      SnapshotOptions:
        AutomatedSnapshotStartHour: 0
      VPCOptions:
        SubnetIds:
          - !Ref PublicSubnet1
          - !Ref PublicSubnet2
        SecurityGroupIds:
          - !GetAtt ElasticsearchSecurityGroup.GroupId
          - !GetAtt ReceiveFromElasticsearchSecurityGroup.GroupId
          - !GetAtt ReceiveFromBastionSecurityGroup.GroupId
      AccessPolicies:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal: '*'
            Action: es:*
            Resource: !Sub 'arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ElasticsearchName}/*'
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}-EsDomain'

  ElasticsearchSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: >-
        Identifies resources, which should have access to one another.
      VpcId: !Ref VPC
      SecurityGroupEgress:
        - IpProtocol: -1
          FromPort: -1
          ToPort: -1
          CidrIp: 0.0.0.0/0
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}-EsSg'

  ReceiveFromElasticsearchSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: >-
        Allows ingress from ElasticsearchSecurityGroup.
      VpcId:  !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: -1
          FromPort: -1
          ToPort: -1
          SourceSecurityGroupId: !GetAtt ElasticsearchSecurityGroup.GroupId
      Tags:
        - Key: Name
          Value: !Sub '${AWS::StackName}-ReceiveFromEsSg'

Outputs:
  DomainArn:
    Value: !GetAtt ElasticsearchDomain.DomainArn
    Export:
      Name: !Sub '${AWS::StackName}-EsDomainArn'

  DomainEndpoint:
    Value: !GetAtt ElasticsearchDomain.DomainEndpoint
    Export:
      Name: !Sub '${AWS::StackName}-EsDomainEndpoint'

  BastionIp:
    Value: !Ref BastionElasticIp
    Export:
      Name: !Sub '${AWS::StackName}-BastionIp'

  ElasticsearchSecurityGroupId:
    Value: !GetAtt ElasticsearchSecurityGroup.GroupId
    Export:
      Name: !Sub '${AWS::StackName}-EsSgId'

  ReceiveFromElasticsearchSecurityGroupId:
    Value: !GetAtt ReceiveFromElasticsearchSecurityGroup.GroupId
    Export:
      Name: !Sub '${AWS::StackName}-ReceiveFromElasticsearchSgId'

  BastionSecurityGroupId:
    Value: !GetAtt BastionSecurityGroup.GroupId
    Export:
      Name: !Sub '${AWS::StackName}-BastionSgId'

  ReceiveFromBastionSecurityGroupId:
    Value: !GetAtt ReceiveFromBastionSecurityGroup.GroupId
    Export:
      Name: !Sub '${AWS::StackName}-ReceiveFromBastionSgId'

  PublicSubnetId1:
    Value: !Ref PublicSubnet1
    Export:
      Name: !Sub '${AWS::StackName}-PublicSubnet1'

  PublicSubnetId2:
    Value: !Ref PublicSubnet2
    Export:
      Name: !Sub '${AWS::StackName}-PublicSubnet2'

How to deploy this AWS Cloudformation Stack

Create an IAM Service Role

1. Create a AWS IAM Cloudformation Service Role with extended session duration. Using the AWS IAM console, create a Cloudformation Service Role named 'CloudformationCreateElasticsearchServiceStackServiceRole' (or something like that) with the following permissions:

• AmazonEC2FullAccess
• AmazonVPCFullAccess
• IAMFullAccess
• AmazonESFullAccess
• AmazonEC2ContainerRegistryFullAccess
• CloudWatchFullAccess
• AmazonSNSFullAccess

2. Important: Navigate to the Service Role using the AWS IAM Console, and increase "Maximum CLI/API session duration" to 2 hours or more. Otherwise, you will run into sporadic stack create timeout errors!

Create an EC2 Key Pair for the bastion server

1. Using the AWS EC2 Console, create an AWS EC2 Key Pair for accessing the bastion server. Name it 'my-bastion-key-test-0' or something like that.

Deploy the Stack

1. Ensure your AWS IAM User has sufficient permissions to deploy this stack.

2. Deploy the Elasticsearch Domain stack with the AWS CLI:

aws cloudformation create-stack \
    --stack-name MyElasticsearchServiceTestXyz \
    --template-body file://./elasticsearch-service-template.yaml \
    --tags Key=Environment,Value=Test Key=Project,Value=MyElasticsearchProject \
    --timeout-in-minutes 120 \
    --capabilities \
        CAPABILITY_NAMED_IAM \
        CAPABILITY_IAM \
        CAPABILITY_AUTO_EXPAND \
    --role-arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/CloudformationCreateElasticsearchServiceStackServiceRole \
    --parameters \
        ParameterKey=ElasticsearchName,ParameterValue=my-es-service-xyz-test-0 \
        ParameterKey=BastionKeyName,ParameterValue=my-bastion-key-test-0 \
    # for debugging the stack use `--disable-rollback`