AWS Essential Training

AWS_Notes.md

AWS Essential Training

Aqui

The Cloud:

• Iaas
• Paas
• Saas

Managed w/o Cloud Services:

• Application Data and Data
• Runtime and OS
• Servers, Load Balancers, File Storage, Networking

SaaS manages everything

Elasticache - distributed memory cache

AWS RDS - distributes data

Best Practices

Design for Failure

No single points of failure

Move DB to its own instance, load balancer, multiple web servers

Implement Eslasticity

• You can scale on a daily basis

• Scaling in anticipation of an event

• Set up network metric monitoring

• Bootstrapping, a self-sustaining deployment process

Loose Coupling

3-Tier Architecture: Load Balancers (or message queue) + App Servers + Web Servers + Database layers (layers are not aware of each other)

Optimizing For Performance

S3 and Cloudfront use serverless architecture, Lambda eliminates need for app servers

Security

AWS Customer is response for network and pp level security as well as data and data in motion.

AWS provides encryption options (EBS - Elastic Block Storage)

AWS Access Keys (public and secret)

Use https for AWS API endpoints

Pass in access key during app build

Use IAM roles

Rotate the keys often

Cost

Embrace consumption payment model (pay as you go)

Turn off test servers at night.

Auto Scaling

Bill reports

Virtual SErvers, EC2, and Elastic IP

EC2 (Elastic Cloud Computer)

Elastic IPs are created at the account level, the account controls access, makes it easily remappable. If an IP server fails, the IP address can be remapped to new server.

Can also be used for version control and regional collections

There is no data replication among regions

Most system requirements can be acheived in just one region.

Regions contain at least 2 availability zones, these can be made redundant.

AMI Amazon Machine Image

Packaged environemnt and setting to boot up EC2 instance, deploy one or several

Template for the root volume

Launch permissions

Block device mapping

Responsible for setting up and booting up instances.

Elastic Load Balancing

Characteristics:

• handles http, https, and tcp
• supports health checks
• dynamically grows and shrings
• single CNAME for DNS configuration, adds IP addresses as use increases

CloudWatch

Monitors resources (no installation necessary)

Paid plans make more detailed monitoring available.

EBS - Elastic Block Storage

EBS is persistent, attached to EC2 instances

Standard

Provisioned IOPS volumes

Point-in-time snapshots can be created, copy snapshots across regions

Allows volumes and data to be shared.

Snapshots are stored incrementally

Only bulled for amount of data changed. Deleting removes what is not neede by other snapshots.

EBS snapshots can be accessed immediately

lazy loads data to new volumes

Snapshots help resize volumes

Keeps persistent data independent of EBS instances.

RDS Relational DB Service

Scalable

East of management

Fast disaster recovery

Create an RDS using API or web console

Specify database software (MySQL, Postergres, etc)

Cloudwatch can monitor the general health

No SSH or root access - Amazon is in charge of security patches and backups.

High availability and fault tolerance

Read replicas make DB more performant

Bootstrapping

A sustainable startup process that can work on its own.

Mount the drives - start services - update files - get latest versions of 3rd party software

Tools: custom scripts, Cloud-Init (Linux), EC2Config (Windows)

Others: AWS Elastic BeanStalk, OpdWorks, Cloud Formation, CodeDeploy

Autoscaling

You define scale out and scale in params

3 components - 1. Launch config (what to scale) 2. Autoscaling group (where to launch instances and defines limits) 3. Scaling Policy (conditions for instant scaling (optional))

Scalable Storage

• EFS - Files

• EBS and EC2 - block storage

• S3 - object store (accessedby keys), any blob of adata (images, videos, etc.)

Objects are stored redundantly

RRS - reduced redundancy storage (cost savings measure)

5TB limit on each size, cna be accesed via REST api

Okay for consistent data (backups, etc.) not good for constantly changing data (takes a while to propogate deletions for example)

Glacier objects are good for backups

Elasticity

AWS Beanstalk - very convenient, fully managed, manages resource provisioning, scaling, and balancing automatically (supports IIS for .NET applications)

Workflow:

Write your code -> Save and name a code version -> Upload to Amazon S3 bucket (can use git deployment AWS Mgmt Console, or AWS Toolkit for VS) -> Tell Beanstalk to deploy it -> Deploy or roll back as needed

AWS Opsworks Stacks

Stack is a set of servers that solve a certain problem. Allows more granular control.

Provides provisioning engine, launches instances, attaches volumes, configures ELBs and handles auto helaing

Workflow:

Stack created -> layers created -> application defined -> check recipes written -> instances launched -> application deployed

Cloud Formation

Easily manage a collection fo resources

The entire application is managed as a series of text-based template files.

Easily version controlled and replicated

Cloud Former Workflow:

Configure and laucnh required resources -> create and launch CloudFormer stack -> create a template -> shut down cloud formation stack -> use the cloud former template as needed

Code Deploy

It's a component service, coordinates deployments across EC2 instances, not framework, language, or stasck dependent, and it's scalable

Code Files and App Spec files are created -> Sent to AWS S3 bucket -> Specify revision and deployment group -> agents poll CodeDeploy to determine what to do.

Simple Message Queue Service (SQS)

Improves scalability and survives single component failure.

Components use queue buffers to communicate, handles spikes and buffers.

Watermarking App Example: (* means queue*)

User -> Webserver -> uploaded message -> resizer (x2) -> Ready for watermark message -> Watermarker(x2) -> ** read for notification message* -> Notifier (x2)

SQS is the glue that binds independent components together in a system

Amazon SWF

(Simple Workflow)

Delivers boilerplate code

Activity workers perform the work assigned by SWF tasks. Work is decided by the decider.

Every task is logged in the log bucket, then sent to SWF, results returned to SWF after each task.

Simplifies complex, distributed, and parallel workflows

Amazon SNS

Pub-Sub Protocol: same message is sent to multiple subscribers

Can be sent through different protocols

Push rather than pull (SQS polls rather than pushes)

Amazon Dynamo DB

NoSQL DB

Scales automatically, stored on solid-state drives, automatically replicates, helps with maintaining stateless servers

Ex. an app is distributed to two different servers, if sessions state is stored on one, the other server doesn't know if user is authenticated in or not.

Sticky session management can store routing information on the load balancer, and route all requests for one user to one server, but this also itnroduces single point of failure.

Anotehr solution

User -> Load balancer -> Servers (times n) -> DynamoDb (Stores session)

Keep application components as stateless as possible

Caching: AWS Elasticache

Caching layer should be as close to DB layer as possible, with elasticache, caching is synched among webservers so caching is optimized and in sync.

Supports MemCacheD and Redis

MemcacheD -> string datatype, up to 1MB value, no persistence, easy to scale

Redis -> bigger string values, 512 MB, persistent, Lua scripting can use custom commands, other data types

Write-through pattern: data is written to DB first, then written to cache. Increases cache hits, but more memory needed.

Lazy Load Patern: Attempts to retrieve first, if not in cache, looks to DB.

TTL Time to Live Settings: uses both approaches, expires data after a period of time.

Amazon CloudFront

CDN service from Amazon. Ensures more data kept in client's cache.

2 components: a distribution and edge locations (network of caches)

Requests get routed to Edge Cache first, then Regional Cache if not found, then Origin server if still not found. This reduces latency.

AWS Cloud Search

DBs make bad search engines. Cloud search is structured data + specific queries that make results more relevant to users.

A search domain makes all data structured -> made of document endpoint and search endpoint

Define nd configure search fields, names of fieds are a schema for search query

Upload data, then run search queries.

Made of 3 services: config, doc and search

Text analysis schemes find stop words and synonyms, stem mapping (fish also returns fishing). These are customizable. Suggesters find possible answers for incomplete search query.

Retrieve data -> convert data to something CloudSearch understands, removing bad characters -> Create the document which represents the search result -> upload doc to search endpoint

AWS API Gateway

Helps with building API with versioning, caching, throttling, scaling, security, auth, and monitoring.

Gateway helps by executing application code, usually on an EC2 instance. Can be cahced, or could be denied if not authed or part of DOS attack.

Request -> checks max calls -> checks throttling limits -> checks cache -> sends to backend

Can manage versions and release stages, automatic SDK generation.

Virtual Servers

Abtracts the hardware, EC2

Containers abstract the OS (ECS and Elastic Beanstalk)

Serverless -> language runtime is abstracted out (AWS lambda)

Lambda Components = function itself + event source

Security: the Shared Security Model

Amazon provides Security of the Cloud, Customer is repsonsible for Security in the Cloud

Categories of service:

1. Infrastructure Services - EC2, EBS, Autoscaling
2. Container Services - EC2 w/ OS layer abstracted out, RDS, ES, E Beantalk
3. Software Services - Elastic Transcoder, Lex, Athena

IAM

Account, access, and roles configuration (Identity and Access Mgmt)

Master Account is set up by initial first user (liked root on a nix system) This shouldn't be used for routine access, just config.

Don't share creds, turn on MFA. Users, groups, roles, permissions

Security principal of least priviledge, new users have no access by default.

Permissions can also be set at group level.

Roles are assumed by resources requiring role, can be compartmentalized.

Security Groups

Control inbound and outbound traffic to a resource.

Traffic type: SSH, RDP, HTTPS

Protocol: TCP, UDP

Port ranges: IP or other security group

Uses CIDR notation

Example: User (:80) -> Web Server (:6081) -> App Server (port :3306) -> Database Server

User -> Firewall -> inbound on port :80 to Web Server ->

Routing done on Bastion Server, only accessible via ssh port :22 ->

App Server connects to DB on port :3306 ->

all other ports blocked

Virtual Private Cloud (VPC)

Instances are isolated to one AWS account.

Not addressable in public internet

Private and public interface control

Elastic networks can be assigned

VPN Connection

Default VPCs are available w/o configuration

VPC Setup:

• Chooser region
• range of IP addresses are assigned
• ensure range allows for growth
• Subnet, IP rance of entire VPC network
• defined within availability zones

Optimizing for cost

Monthly expenses can be estimated using AWS calculator, cost centers can be calculated, reports can be generated.

App Development

First, create an IAM user instead of signing in as root.

Key Pairs:

Found under EC2 instance: Left Panel -> Netowrk and Security -> Key Pairs

Choose button Create Key Pair (name should follow a convention and key pairs should be rotated out)

This will automatically download a PEM file.

Put PEM where it goes, then restrict permissions to readonly:

$ chmod 0400 my-pem.pem

Creating a security group

Found under EC2 instance: Left Panel -> Netowrk and Security -> Security Groups

Click 'Create Security Group'

Create load balancer (accepts http port :80 and https prt :443).

Copy the Group Id of load balancer group

Create web tier group -> traffic should be http then copy group Id of load balancer for traffic that should be accepted.

Repeat for database tier

Creating an EC2 Instance

From EC2 dash, clikc Launch Instance

Select AMI (Amazon Linux or something else)

Select size

Configure Instance Details: Select Zone 2a if you're using load balancer

At the bottom, under advanced, you can paste in Bash scripts. The following sets up LAMP server:

#! /bin/bash -ex
yum update -y
yum groupinstall -y "Web Server" "MySQL Database" "PHP Support"
service httpd start
chkconfig httpd on
groupadd www
usermod -a -G www ec2-user
chown -R root:www /var/www
chmod 2775 /var/www
find /var/www -type d -exec chmod 2775 {} +
find /var/www/ -type f -exec chmod 0664 {} +

Add Storage (If Necessary)

Add Tag:

Tags can add names and environment configurations. For ours, let's use:

Name : web-server-1 env : test

Configure Security Group

Select web-tier security group (from previous section) and click launch. Select previous key-pair to associate with this instance.

Configure Load Balancer

Select Load Balancing -> Load Balancers

Select between Application Load Balancer and Classic Load Balancer (Choose Application)

Name it.

Add listeners => http port:80 and https port:443

If you add https, Amazon will walk you through generating a certificate.

Choose availability zones.

Configure security (select load balancer group from previous section)

Configure Routing

Register Targets -> add web server

Accessing Web Server via load balancer

When accessing the site now, it returns a 403 (or 502)

Connect to EC2 Instance

Go to Instances -> Instances, select your instance and copy the IPv6 Public IP

Where your PEM is located, enter: $ ssh -i my-pem.pem ec2-user@[YOUR IPv6 IP]

Then you'll be on the instance.

Creating a MySql RDS Database

Search : RDS -> Create DB ->

Step 1 : Select Engine (MySQL or Aurora)

Step 2: Use Case (Dev/Test is simplest, cheapest option)

Step 3: Choose instance specifications (micro is simplest) (no multi-AZ deployment) (20 GB storage (min))

Step 4: Advanced Settings

• default VPC
• subnet group default
• availability zone should match EC2 instance
• DB Security Group - select the configured db security
• the rest of the default settings should be fine Launch creates the DB - Instances can be monitored, After DB is ready, endpoint can be copied

Go back to SSH in command line

Check for mySQL: $ which mysql

set up mysql : $ mysql -h [DB ENDPOINT URL] -u admin -p (this will prompt you for the admin password)

MySQL monitor should open which means the db tier is connected.

AMI

Go back to EC2 dash, go to instance then menu Actions -> Image -> Create Image

Name it and provide a description. (No reboot ensures more consistent image copying)

Clicking on the AMI menu will show AMI status

Note the zone availability of the EC2 instance, and the availabilities of the load balancers

Go back to AMI instance, click Actions -> Launch. Settings should be default, but under subnet, make sure you launch into 2nd availability zone of the load balancer (not the one the 1st EC2 instance is in).

Add tags if necessary

Security Group : Should be launched into Web Tier (same as EC2 instance)

Once the instance state is running, we need to register it.

Go to Target Groups. Below, click Targets -> Edit

Select new instance, then add to target. Once initialized, it should balance traffic between both instances equally.

Auto Scaling

From console, go to EC2 -> Autoscaling(from menu) -> Autoscaling Group -> Create AutoScaling Group

Choose an AMI for new servers. Previous AMI can be found under My AMIs. Select it.

Select instance type -> name it -> storage (if necessary) -> Security Group: Web Tier

Create Autoscaling Group: name it, group size should be 2 ofr both EC2 instances,

For subnet, include both availability zones from the load balancer

Under Advanced,

Enable load balancing

Select target group from load balancer

Scaling policies.

Notifications, Tags, etc.

Elastic Beanstalk

Abstracts out EC2 setup, automating what we di previously

Serverless Architecture