adhorn

1. Within the ASG1 that runs the service you want to update, find the number of instances required to serve the current traffic.
2. Start a new ASG2 with instances running the new version of the service. This should be a new golden AMI.
3. Launch the ASG2 with the same number of instances found in step 1 (+ typical 5-10%)
4. At this point, the new ASG2 is taking traffic along side ASG1.
5. De-attach the ASG1 from the ELB but don't stop the instances quite yet. This is your ticket to rollback.
6. If all looks good after few hours, you can dispose of ASG1.

adhorn

Traffic Shifting Using an Alias (CLI)

To configure an alias to shift traffic between two function versions based on weights. 
Here is an example that points an alias to two different Lambda function versions, starting with version 1 receiving 90% and version 2 receiving 10%

> aws lambda create-alias --name myalias --function-name myfunction --function-version 1 --routing-config AdditionalVersionWeights={"2"=0.1}

Then you can update the percentage of incoming traffic to version 2, here to 15%.

> aws lambda update-alias --name myalias --function-name myfunction --routing-config AdditionalVersionWeights={"2"=0.15}

adhorn

(1) version 1 receiving 90% and version 2 receiving 10%
> aws lambda create-alias --name myalias --function-name myfunction --function-version 1 --routing-config AdditionalVersionWeights={"2"=0.1}

(2) you can update the percentage of incoming traffic to version 2, here to 15%.
> aws lambda update-alias --name myalias --function-name myfunction --routing-config AdditionalVersionWeights={"2"=0.15}

Eventually, after several cycle of incremental increase.

(n) you can route all traffic to version 2 by changing the function-version property to point to version 2 and set the routing-config parameter to an empty string, as shown following.
> aws lambda update-alias --name myalias --function-name myfunction --function-version 2 --routing-config '' 

adhorn

def cust_fun():
    print("Hello from the deep layers!!:")
    return 1

adhorn

from __future__ import division, unicode_literals

import sys
sys.path.insert(0, '/opt/python/.vendor')

from ssm_cache import SSMParameter
import time
import random
import json
import requests

adhorn

from __future__ import unicode_literals
import sys
import logging
import boto3
import os

log = logging.getLogger()
log.setLevel(logging.DEBUG)

sys.path.insert(0, './vendor')

adhorn

def get_config(config_key):
    param = SSMParameter(os.environ['FAILURE_INJECTION_PARAM'])
    try:
        value = json.loads(param.value)
        if not value["isEnabled"]:
            return 0, 1
        return value[config_key], value.get('rate', 1)
    except InvalidParameterError as e:
        # key does not exist in SSM
        raise InvalidParameterError("{} does not exist in SSM".format(e))

adhorn

service: ChaosInjectionLayer

frameworkVersion: ">=1.34.0 <2.0.0"

provider:
  name: aws

layers:
  latencyInjection:
    path: ./

adhorn

def corrupt_delay(func):
    def latency(*args, **kw):
        delay, rate = get_config('delay')
        if not delay:
            return func(*args, **kw)
        print("delay: {0}, rate: {1}".format(delay, rate))
        # if delay and rate exist, delaying with that value at that rate
        start = time.time()
        if delay > 0 and rate >= 0:

adhorn

def corrupt_exception(func):
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        exception_msg, rate = get_config('exception_msg')
        if not exception_msg:
            return result
        print("exception_msg from config {0} with a rate of {1}".format(exception_msg, rate))
        # add injection approx rate% of the time
        if random.random() <= rate:
            print("corrupting now")