Skip to content

Instantly share code, notes, and snippets.

@nicolasdao

nicolasdao/elasticbeanstalk_guide.md

Last active December 11, 2023 14:40
Show Gist options
  • Save nicolasdao/ac7027a9cf59e1f39bb5c96193ed4435 to your computer and use it in GitHub Desktop.
Save nicolasdao/ac7027a9cf59e1f39bb5c96193ed4435 to your computer and use it in GitHub Desktop.
Download ZIP
Elastic Beanstalk guide. Keywords: elastic bean beanstalk elasticbeanstalk cli eb eb
Raw
elasticbeanstalk_guide.md

ELASTIC BEANSTALK GUIDE

Table of contents

About this document

This document introduces the use of the EB CLI (Elastic Beanstalk CLI) to manage the lifecycles of applications deployed on AWS Elastic Beanstalk. The standard usage is described for a nodeJS application.

Installing the EB CLI

Prerequisite:

  • Python 2 version 2.7, or Python 3 version
  • AWS CLI. That's technically not required, but it makes things easier. EB CLI relies on the same credentials files managed by the AWS CLI. To install the AWS CLI, please refer to my AWS CLI Guide.

Hopefully, you're using a MAC (for Windows users, good luck). For MAC:

brew update
brew install awsebcli

The EB CLI uses the same credentials than the AWS CLI, so make sure that you have those credentials set up on your machine. To set that up, please refer to my AWS CLI Guide.

Concepts - Application vs Environment vs Version

A version is the actual code that is deployed to EB. There is no need to specify it explicitely. Each time the eb deploy command is executed, a new version is automatically created. To deploy code to EB, an application with an environment must exist. An application can have an arbitrary number of environments (e.g., dev, test, prod). Each of those environment can be configured differently.

Getting started

TL;DR

If you're after the step-by-step manual to setup your NodeJS project in EBS, then refer to the Pragmatic setup in the Annexes.

  1. Create your local project the way you would normally do it (e.g., your typical NodeJS server).

    WARNING: If you're deploying a web server, don't forget to listen to the correct port. The correct port is accessible via the environment variable PORT (e.g., NodeJS Express: app.listen(process.env.PORT)).

  2. Create the application in AWS and initialize the project:

    eb init --profile <your-profile-name>

  3. Edit the branch-defaults.master.environment property in the .elasticbeanstalk/config.yml file (set it to something like my-app-test, my-app-staging or my-app-prod. The reason why you should prefix the environment with your something similar to my-app is explained in the Step 4 - Create an environment section).

  4. Create a new .ebextensions folder and add the necessary .config files to setup your stack (ref. to Step 2 - Stack configuration).

  5. Create a new .ebignore to disconnect Git (copy/paste the content of the .gitignore in it).

  6. Create a new environment in AWS:

    eb create <environment-name> --cname <cname>

    where:

    • <environment-name> is the same as the environment set in step #3.
    • <cname> must be a unique name that is used to create a unique URL as follow: <cname>.<region>.elasticbeanstalk.com

  7. Later down the track, when you have new changes to deploy, use:

    eb deploy

Before we start you must know about EB CLI and Git

The EB CLI has a strongly opiniated way to work with Git that you must be aware of if you don't want to pull your hair out wondering why your deployments don't seem to do anything.

By default, when an application is using Git, only committed changed are deployed. The typical issue occurs when you've added a lot of changed to your app, but you've forgotten to execute git commit -am 'your new changes description' before executing eb deploy.

There are two ways to override this default behavior:

  1. Use eb deploy --staged. This command deploys the changes in the staged tree rather than the head. This allows to not be forced to commit the changes before deploying, but it still requires to add the latest changes to the staged tree (git add .). Because I mainly uses nodejs, my suggestions to nodejs readers is to leverage npm scripts to automate the deployments. For example, add something similar to this in your package.json:

    "scripts": {
  "deploy": "git add . & eb deploy --staged"
}

    With this, you can use the command npm run deploy with the peace of mind that what you see in your text editor is what's being deployed.

  2. Add an .ebignore file. Adding this file under the application root directory breaks the connection between git and the deployment command. Copy/paste what is normally in the .gitignore, so you can stop to worrying about what has been commited or not before a deployment.

Step 1 - Application creation & project initialization

This step serves two purposes:

  1. It initializes your local project with a new .elasticbeanstalk folder containing a config.yml (see example under The .elasticbeanstalk/config.yml section).
  2. It creates the Elastic Beanstalk application in your AWS account. That's why it requires a valid AWS profile set up for that account on your local machine.

To complete this step, run the following command:

eb init --profile <your-profile-name>

This command contains some smart to detect the project stack (e.g., PHP, nodsejs). If it does not, pleae refer to the list of supported stack here.

When this command is completed, edit the .elasticbeanstalk/config.yml and set the branch-defaults.master.environment property to the whatever name makes sense to you (personally, I like using the application name followed by '-test' or '-prod'. I also use that name for the CNAME).

Step 2 - Stack configuration

This step consists in configuring aspects of the application such as autoscaling, environment variable as well as settings specific to your stack. Depending on the application stack, there might be different tasks that have to run before starting the application. For example, to use a nodeJS server, the stack might need to start it using npm start, otherwise, the application might not be able to respond though it has been successfully deployed (this usually results in a 502 Bad Gateway error message from the ngninx load balancer). To configure the extra steps:

  1. Create a new .ebextensions folder under the application root directory.
  2. Add as many .config files as you need to configure the application (the name of the files does not matter. eb CLI processed them all. What matters is their content). Choosing between one or many .config file depends on your personal preferences or requirements.
  3. Configure the .config files. All of them must be structured as follow: 
    option_settings:
  - namespace: This might be optional. It depends on the setting.
	option_name: Name of your option.
	value: Option value.
  - namespace: Another one
	option_name: Another one
	value: Option Another one

There are two types of option settings:

  1. General options: Those options helps configure EB itself (scaling, environment variables, security groups, ...). The list of all the options can be found here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options-general.html. The following is shows how to set up an environment variable as well as the autoscaling:

    option_settings:
  - namespace: aws:elasticbeanstalk:application:environment
	option_name: HELLO
	value: WORLD
  - namespace: aws:autoscaling:asg
	option_name: MinSize
	value: 1
  - namespace: aws:autoscaling:asg
	option_name: MaxSize
	value: 2

    Alternatively, this .config file can be broken down using a slightly different API:

    austoscaling.config:

    option_settings:
  aws:autoscaling:asg:
	MinSize: 1
	MaxSize: 1

    envvar.config:

    option_settings:
  aws:elasticbeanstalk:application:environment:
	HELLO: WORLD

  2. Stack specific options: Based on the stack you've deployed, there might be specific options required to configure it. You can find all of them at https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options-specific.html. For example, an Express NodeJS app that is not using an app.js or a server.js as its starting script needs a start script in its package.json similar to this:

    "scripts": {
  "start": "NODE_ENV=production node index.js"
}

    Where index.js is the file containing the JS code starting your Express server.

    IMPORTANT: IGNORE THE NEXT LINE IF YOU'RE USING LINUX 2 OS. JUMP TO THE LINUX 2 OS SECTION BELOW:

    In that case, you have to add under the .ebextensions folder a .config file similar to this:

    whatevernameyoulike.config

    option_settings:
  - namespace: aws:elasticbeanstalk:container:nodejs
	option_name: NodeCommand
	value: "npm start"

    LINUX 2 (RECOMMENDED): With the latest OS, the aws:elasticbeanstalk:container:nodejs option is not supported anymore. This is replaced with the following process:

    1. Create a new Procfile file in your project's root.
    2. Add the following line: web: npm start

Step 3 - Configure Git

As explained in the previous section Before we start you must know about EB CLI and Git, Git plays an important role

Step 4 - Create an environment

An environment is a contained set of resources (e.g., load balancer, security group, EC2 instances) needed to host your app. There can be an arbitrary amount of environments (e.g., dev, test, prod). An application can be deployed if there is not at least one environment.

To create an environment, make sure you've followed this step's prerequisites and then run the following command:

eb create <environment-name> --cname <cname> --elb-type <load-balancer-type>

Where:

  • <environment-name> is the same as what has been set up in the branch-defaults.master.environment property in the .elasticbeanstalk/config.yml file. Set it to something like my-app-test, my-app-staging or my-app-prod. The reason why we recommmend you prefix the environment with something similar to my-app is because that label is also used to name the EC2 instances. This will become all very confusing when you list your EC2 instances.
  • <cname> is used to create the HTTP endpoint for the load balancer. That endpoint is follows this schema: cname.aws-region.elasticbeanstalk.com. This cname must be globally unique.
  • --elb-type <load-balancer-type> changes the default load balancer (classic, application (default), network). This is optional.

For a list of all the available options to create a new environment, please refer to https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-create.html#eb3-createoptions.

WARNING: This step is the ONLY CHANGE YOU HAVE to choose which load balancer you need (classic, application (default), network). Once you've choosen that LB type, you can't change it. However, you can change its configuration during each deployment.

To see how to create environments on existing VPC please refer to section How to create an environment with an existing VPC?

Step 5 - Deploy

This step is automatically done after the environment creation (via eb create), so you only have to use it for subsequent changes. Use this command to start a new deployment:

eb deploy

IMPORTANT: If you're using git, this only deploys the last commited changes, unless you're using a .ebignore file as described in Before we start you must know about EB CLI and Git.

To deploy the changes that are in the staged tree, use:

eb deploy --staged

TIPS: To avoid missing changes, add scripts to automate your deployments as described previously in the Before we start you must know about EB CLI and Git section.

Those commands deploy the application to the current environment in the current account.

Step 6 - Change the configuration

This step is optional. If you need to change the configuration (e.g., the min or max number of instances), sue this command:

eb config

The .elasticbeanstalk/config.yml

---
  branch-defaults: 
    master: 
      environment: "your-app-name-test"
      group_suffix: null
  global: 
    application_name: "your-app-name"
    branch: null
    default_ec2_keyname: null
    default_platform: "64bit Amazon Linux 2 v5.6.4 running Node.js 14"
    default_region: "ap-southeast-2"
    include_git_submodules: true
    instance_profile: null
    platform_name: null
    platform_version: null
    profile: "your-awscli-profile-name"
    repository: null
    sc: "git"
    workspace_type: "Application"

Notice:

  • default_platform: "64bit Amazon Linux 2 v5.6.4 running Node.js 14". This sets up the underlying OS. To list all the available platforms, use this command: aws elasticbeanstalk list-available-solution-stacks

Using CloudFormation

Though it is possible to use CloudFormation to manage an EB stack, that can be quite challenging due to the fact that the deployment of new code changes should also be manage by CF. Indeed, if CF is used to set up the EB stack and the EB CLI use subsequently used to deploy code, then any other changes made with CF will overide all changes made with the Eb CLI. That's why I personaly prefer to stick with the EB CLI to manage the lifecycles of the EB stack.

In any case, the next section shows how to create a CF template to create a nodeJS EB stack.

The template

AWSTemplateFormatVersion: '2010-09-09'

Description:
  Sets of resources required to provision your application.

Parameters:
  Environment:
	Description: 'The operating environment'
	Type: String
	Default: dev
	AllowedValues: [ dev, prod ]
  ApplicationName:
	Type: String
	Default: my-super-app

Conditions:
  isProd: !Equals [ !Ref Environment, prod]

Resources:

  yourApplication:
	Type: AWS::ElasticBeanstalk::Application
	Properties:
	  ApplicationName: !Ref ApplicationName
	  Description: AWS Elastic Beanstalk microservice that does something awesome.
  
  yourConfigurationTemplate:
	Type: AWS::ElasticBeanstalk::ConfigurationTemplate
	Properties:
	  ApplicationName: !Ref yourApplication
	  Description: AWS ElasticBeanstalk Sample Configuration Template
	  OptionSettings:
	  - Namespace: aws:autoscaling:asg
		OptionName: MinSize
		Value: '1'
	  - Namespace: aws:autoscaling:asg
		OptionName: MaxSize
		Value: '2'
	  - Namespace: aws:elasticbeanstalk:environment
		OptionName: EnvironmentType
		Value: LoadBalanced
	  SolutionStackName: 64bit Amazon Linux 2018.03 v4.12.0 running Node.js
  
  yourEnvironment:
	Type: AWS::ElasticBeanstalk::Environment
	Properties:
	  ApplicationName: !Ref yourApplication
	  EnvironmentName: !If [isProd, 'prod', 'test']
	  Description: AWS ElasticBeanstalk application environment
	  CNAMEPrefix : !Join [ '-', [ !Ref ApplicationName, !Ref Environment ] ]
	  TemplateName: !Ref yourConfigurationTemplate

A list of all the supported solution stack name (SolutionStackName) can be found Elastic Beanstalk Supported Platforms.

To deploy this template, use the AWS CLI with this command:

aws cloudformation deploy --profile <your-profile-name> --template-file <path-to-template>.yml --stack-name <your-stack-name> --capabilities CAPABILITY_IAM --parameter-overrides Environment=staging",

Blue/Green deployments

Original doc: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.CNAMESwap.html

  1. Create a new environement using the same project but a different name.
  2. Upon creation, check that everything is working fine.
  3. Swap URLs.

Troubleshooting

Health is marked as severe

This is either a serious or benign issue:

  • Benign issue: This often happens when your healthcheck endpoint is badly configured. By default, the healthcheck is hosted on port 80 on pathname /. If you have no endpoint listening on that port and path, then AWS EBS will infer your API is in severe state because no traffic can reach it. If that's the case, simply create an echo endpoint on port 80 for pathname /. If on the other hand, you do have a dedicated healthcheck endpoint which is not hosted on port 80 for pathname /, then you must configure that endpoint in your AWS EBS configuration.
  • Serious issue. Your app might be down. Test it manually to confirm. If that's the case, check the logs to figure out what's going on.

502 Bad Gateway

This error message appears in your browser when you're hitting your Web App.

Potential reasons:

  • Check that your web server is configured so it starts. For example, in NodeJS, you need to add a config so that the npm start command is executed when the server starts (ref. to Step 2 - Stack configuration for more details).
  • Your web server is not listening on the right port. The right HTTP port listener is defined in the environment variable PORT (e.g., with an Express NodeJS server, use this code snippet to start the server: app.listen(process.env.PORT)).

If none of those reasons can explain this issue, download the logs via the AWS Console.

Cannot upgrade the OS aka platform version

This configuation is maintained in the .elasticbeanstalk/config.yml file under the default_platform property. If the OS upgrade is too far ahead of the current version, you must proceed to a Blue/Green deployments.

Conclusion

Google App Engine vs AWS Elastic Beanstalk

Features/Facets App Engine Elastic Beanstalk
Ease of use Very easy Easy to painful
Scalability High High
HTTPS Out-of-the-box with no config Have to be manually configured. SSL must be provided
Can scale to 0 Yes No

FAQ

Where is

Where can I find a list of all the configurable options?

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-options-general.html

Where do I configure the platform branch?

Use the default_platform property under the .elasticbeanstalk/config.yml file.

How to

How to setup environment variables?

Add a new .config file under the .ebextensions folder similar to this:

option_settings:
  aws:elasticbeanstalk:application:environment:
	HELLO: world
	HELL: yeah

To use environment variable locally, I would recommend to use shell script similar to this:

#!/bin/sh

export HELLO=world
export HELL=yeah

And then use something like source env.sh.

How to send all logs to CloudWatch?

Add the following .config file under the .ebextensions folder:

option_settings:
  aws:elasticbeanstalk:cloudwatch:logs:
	StreamLogs: true

How to configure the max and min number of instances?

eb config

Under the aws:autoscaling:asg section, update one of the following config:

  • MaxSize
  • MinSize

How to setup the load balancer's ports?

Add a new .config file under the .ebextensions folder similar to this:

option_settings:
  aws:elb:listener:
	ListenerProtocol: TCP
	InstancePort: 8081
  aws:elb:listener:3000:
	ListenerProtocol: TCP
	InstancePort: 8081

This config setup the default port (i.e., 80) to redirect TCP traffic onto port 8081 on the EC2 instance as well as port 3000.

How to create an environment with an existing VPC?

Let's assume the following:

  • The VPC is called vpc-a and it is located Sydney which has only two AZs.
  • There are four existing subnets :
    1. subnet-private-a for AZ-1
    2. subnet-private-b for AZ-2
    3. subnet-public-a for AZ-1
    4. subnet-public-b for AZ-2

Use case 1 - The EB stack can only be accessed privately

This use case covers scenarios where the resources accessible via the EB stack cannot be accessed from the internet (i.e., publicly) for security reasons. Both the load balancer and the EC2 instances live in the private subnets.

eb create <environment-name> --cname <cname> --elb-type application \
  --vpc.id vpc-a \
  --vpc.elbsubnets subnet-private-a,subnet-private-b \
  --vpc.ec2subnets subnet-private-a,subnet-private-b \

Use case 2 - The EB stack can be accessed publicly

eb create <environment-name> --cname <cname> --elb-type application \
  --vpc.id vpc-a \
  --vpc.elbsubnets subnet-public-a,subnet-public-b \
  --vpc.ec2subnets subnet-private-a,subnet-private-b \
  --vpc.elbpublic

Notice that even though the load balancer is in a public subnet, you still need to toggle its public access via the --vpc.elbpublic option.

How to setup specific ingress rules on the EC2 instance?

Add the new ingress rules in the security group that allows traffic between the LB and the EC2 instance by adding a new .config file under the .ebextensions folder. The following example shows how to add public access on port 8081 for both IPv4 and IPv6:

Resources:
  PublicIPv4Traffic:
	Type: AWS::EC2::SecurityGroupIngress
	Properties:
	  GroupId: {"Fn::GetAtt" : ["AWSEBSecurityGroup", "GroupId"]}
	  IpProtocol: tcp
	  FromPort: 8081
	  ToPort: 8081
	  CidrIp: 0.0.0.0/0
  PublicIPv6Traffic:
	Type: AWS::EC2::SecurityGroupIngress
	Properties:
	  GroupId: {"Fn::GetAtt" : ["AWSEBSecurityGroup", "GroupId"]}
	  IpProtocol: tcp
	  FromPort: 8081
	  ToPort: 8081
	  CidrIpv6: ::/0

Annexes

Popular commands

Command Description
eb --help> Show all the available eb CLI's commands.
eb init --profile <your-profile-name> Initializes the project.
eb list Lists all the EB environments created for the current AWS Account(1).
eb create Creates a new environment for the current AWS Account(1).
eb deploy Deploys the latest git commit to the active environment (to switch environment, please refer to command eb use).
eb deploy --staged Deploys the staged git changes to the active environment (to switch environment, please refer to command eb use).
eb terminate Deletes the current environments with all its resources, except the S3 buckets.
eb use <environment name> Switches to another environment.
eb config Configure the current environment.

(1) The current AWS account is defined via the profile field in the .elasticbeanstalk/config.yml.

Supported stacks

  1. Packer Builder
  2. Single Container Docker
  3. Multicontainer Docker
  4. Preconfigured Docker
  5. Go
  6. Java SE
  7. Java with Tomcat
  8. .NET on Windows Server with IIS
  9. Node.js
  10. PHP
  11. Python
  12. Ruby

Pragmatic setup

Overview

The main strategy detailed below consists in not putting the .ebextensions and .elasticbeanstalk folders under source control and to a large extend, not really care about them. Instead, their files are described in environment specific folders (e.g., env/test, env/prod) which are put under source control. Those files overrides the .ebextensions and .elasticbeanstalk before each eb create or eb deploy operations via NPM scripts.

Step-by-step

  1. Create your local Web app normally, and don't forget to set the port properly (e.g., NodeJS: app.listen(process.env.PORT)).
  2. Add the following config to your .gitignore:
# Elastic Beanstalk Files
.ebextensions/*
.elasticbeanstalk/*

The configuration files that should normally be added under those folders are moved into a custom env folder in order to support different configurations that are environment specific (cf. step #6).

  1. Create a new .ebignore file, and copy/paste the content of the .gitignore in it, EXCEPT the extra lines from step #2. We need the content of both the .ebextensions and .elasticbeanstalk to deploy the app.
  2. Create a new Application in AWS with this command: eb init --profile <your-profile-name>
  3. Edit the branch-defaults.master.environment property in the .elasticbeanstalk/config.yml file (e.g., my-app-test) as well as the other fields (refer to the .elasticbeanstalk/config.yml section).
  4. Instead of creating an .ebextensions folder create a new env folder with as many environments folder as you need (e.g., test, prod). In each of those folders:
    • Create the .config files you would have configured under the .ebextensions folder, and make them specific to each environment.
    • Copy/paste the .elasticbeanstalk/config.yml and make it specific to each environment.
    • Create a new env.local file with something similar to:
    #!/bin/sh

export ENV=test
export CNAME=my-app-test
  5. Add a new script in the package.json file as follow: 
    "scripts": {
	"reset-folders": "rm -rf .ebextensions && mkdir .ebextensions || true && rm -rf .elasticbeanstalk && mkdir .elasticbeanstalk || true",
	"setup": "npm run reset-folders && cp -a env/$ENV/*.config .ebextensions && cp -a env/$ENV/*.yml .elasticbeanstalk",
	"create": "npm run setup && eb create $CNAME --cname $CNAME --elb-type application",
	"deploy": "npm run setup && eb deploy",
	"create:test": "source ./env/test/env.local && npm run create",
	"create:prod": "source ./env/prod/env.local && npm run create",
	"deploy:test": "source ./env/test/env.local && npm run deploy",
	"deploy:prod": "source ./env/prod/env.local && npm run deploy"
}
  6. Create a Procfile in your project's root folder as explained in the Step 2 - Stack configuration section.
  7. Create a new environment in AWS or deploy with npm run create:<env> or npm run deploy:<env>.

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment