CloudWatch.log

START RequestId: 9ddba177-f068-4773-bdfa-4064e85d8222 Version: $LATEST
Starting with
context context.Background.WithDeadline(2019-02-11 22:56:29.628991061 +0000 UTC [5.998542519s]).WithValue(&lambdacontext.key
{}
, &lambdacontext.LambdaContext{AwsRequestID:"9ddba177-f068-4773-bdfa-4064e85d8222", InvokedFunctionArn:"arn:aws:lambda:us-east-1:848139458219:function:update-shelters-dev-dbMigration", Identity:lambdacontext.CognitoIdentity{CognitoIdentityID:"", CognitoIdentityPoolID:""}, ClientContext:lambdacontext.ClientContext{Client:lambdacontext.ClientApplication{InstallationID:"", AppTitle:"", AppVersionCode:"", AppPackageName:""}, Env:map[string]string(nil), Custom:map[string]string(nil)}}).WithValue("x-amzn-trace-id", "Root=1-5c61fd8f-e8dc42b140bd56ab87efcb19;Parent=5c01de6a5f6153a2;Sampled=0")
jsonEvent
{
    "RequestType": "Create",
    "RequestId": "bd487606-8017-49f2-99af-b29b2bbad40b",
    "ResponseURL": "https://cloudformation-custom-resource-response-useast1.s3.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A848139458219%3Astack/update-shelters-dev/c08a80e0-2e4e-11e9-87a6-124d1eab42ba%7CSheltersDBMigrationTrigger%7Cbd487606-8017-49f2-99af-b29b2bbad40b?AWSAccessKeyId=AKIAJRWMYHFMH4DNUF2Q&Expires=1549932974&Signature=WlihcQ6TxTy%2FzDrRx9y3t0DAu5k%3D",
    "ResourceType": "Custom::DBMigration",
    "LogicalResourceId": "SheltersDBMigrationTrigger",
    "StackId": "arn:aws:cloudformation:us-east-1:848139458219:stack/update-shelters-dev/c08a80e0-2e4e-11e9-87a6-124d1eab42ba",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:us-east-1:848139458219:function:update-shelters-dev-dbMigration"
    }
}

physicalResourceID: DB_MIGRATION
Connection to database at us1a5sc5mua52k9.c3nckcraddfs.us-east-1.rds.amazonaws.com.
Database connection open to us1a5sc5mua52k9.c3nckcraddfs.us-east-1.rds.amazonaws.com.
Creating schema db_metadata
2019/02/11 22:56:23 Responding with body:
{
    "Status": "SUCCESS",
    "RequestId": "bd487606-8017-49f2-99af-b29b2bbad40b",
    "LogicalResourceId": "SheltersDBMigrationTrigger",
    "StackId": "arn:aws:cloudformation:us-east-1:848139458219:stack/update-shelters-dev/c08a80e0-2e4e-11e9-87a6-124d1eab42ba",
    "PhysicalResourceId": "DB_MIGRATION"
}

2019/02/11 22:56:23 Responding with header:
{}

2019/02/11 22:56:23 Received StatusCode: 200
Received Body: 
END RequestId: 9ddba177-f068-4773-bdfa-4064e85d8222
REPORT RequestId: 9ddba177-f068-4773-bdfa-4064e85d8222	Duration: 169.78 ms	Billed Duration: 200 ms Memory Size: 1024 MB	Max Memory Used: 26 MB	
START RequestId: 9ff22b3d-3d1e-4f37-b98e-9e3ab53fe9e1 Version: $LATEST
Starting with
context context.Background.WithDeadline(2019-02-11 23:00:40.002481456 +0000 UTC [5.996574724s]).WithValue(&lambdacontext.key
{}
, &lambdacontext.LambdaContext{AwsRequestID:"9ff22b3d-3d1e-4f37-b98e-9e3ab53fe9e1", InvokedFunctionArn:"arn:aws:lambda:us-east-1:848139458219:function:update-shelters-dev-dbMigration", Identity:lambdacontext.CognitoIdentity{CognitoIdentityID:"", CognitoIdentityPoolID:""}, ClientContext:lambdacontext.ClientContext{Client:lambdacontext.ClientApplication{InstallationID:"", AppTitle:"", AppVersionCode:"", AppPackageName:""}, Env:map[string]string(nil), Custom:map[string]string(nil)}}).WithValue("x-amzn-trace-id", "Root=1-5c61fe91-083e2c45ac327127bd851339;Parent=48dc9b091aa43896;Sampled=0")
jsonEvent
{
    "RequestType": "Delete",
    "RequestId": "2d166d36-7c0c-4848-9eb5-aedaf5e9172c",
    "ResponseURL": "https://cloudformation-custom-resource-response-useast1.s3.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A848139458219%3Astack/update-shelters-dev/c08a80e0-2e4e-11e9-87a6-124d1eab42ba%7CSheltersDBMigrationTrigger%7C2d166d36-7c0c-4848-9eb5-aedaf5e9172c?AWSAccessKeyId=AKIAJRWMYHFMH4DNUF2Q&Expires=1549933233&Signature=UgQtNEj9vGCvsSOihdJCajKFlsU%3D",
    "ResourceType": "Custom::DBMigration",
    "PhysicalResourceId": "DB_MIGRATION",
    "LogicalResourceId": "SheltersDBMigrationTrigger",
    "StackId": "arn:aws:cloudformation:us-east-1:848139458219:stack/update-shelters-dev/c08a80e0-2e4e-11e9-87a6-124d1eab42ba",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:us-east-1:848139458219:function:update-shelters-dev-dbMigration"
    }
}

Request type is not of type create, returning
physicalResourceID: DB_MIGRATION
2019/02/11 23:00:34 Responding with body:
{
    "Status": "SUCCESS",
    "RequestId": "2d166d36-7c0c-4848-9eb5-aedaf5e9172c",
    "LogicalResourceId": "SheltersDBMigrationTrigger",
    "StackId": "arn:aws:cloudformation:us-east-1:848139458219:stack/update-shelters-dev/c08a80e0-2e4e-11e9-87a6-124d1eab42ba",
    "PhysicalResourceId": "DB_MIGRATION"
}

2019/02/11 23:00:34 Responding with header:
{}

2019/02/11 23:00:34 Received StatusCode: 403
Received Body: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>F7E243255F8FCD3F</RequestId><HostId>8Aok24znlgf5qGYSlF7u0qJRTFjKh6o+v2PlRY8hgIcpME5Fl/t+O4erE6zi3NILoByWpagy3DE=</HostId></Error>
2019/02/11 23:00:34 Received Header:
{
    "Content-Type": [
        "application/xml"
    ],
    "Date": [
        "Mon, 11 Feb 2019 23:00:33 GMT"
    ],
    "Server": [
        "AmazonS3"
    ],
    "X-Amz-Id-2": [
        "8Aok24znlgf5qGYSlF7u0qJRTFjKh6o+v2PlRY8hgIcpME5Fl/t+O4erE6zi3NILoByWpagy3DE="
    ],
    "X-Amz-Request-Id": [
        "F7E243255F8FCD3F"
    ]
}

2019/02/11 23:00:34 Retrying to send with status FAILED
2019/02/11 23:00:34 Responding with body:
{
    "Status": "FAILED",
    "RequestId": "2d166d36-7c0c-4848-9eb5-aedaf5e9172c",
    "LogicalResourceId": "SheltersDBMigrationTrigger",
    "StackId": "arn:aws:cloudformation:us-east-1:848139458219:stack/update-shelters-dev/c08a80e0-2e4e-11e9-87a6-124d1eab42ba",
    "PhysicalResourceId": "DB_MIGRATION"
}

2019/02/11 23:00:34 Responding with header:
{}

2019/02/11 23:00:34 Received StatusCode: 403
Received Body: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>28B4106E0A4F3417</RequestId><HostId>v208XITkl8dU+ePcHuO/upxrpJPiyUGInjO9c03CmLKHTOZFy/BYVQa/Vjm1vT2X0DckbJqehdk=</HostId></Error>
2019/02/11 23:00:34 Received Header:
{
    "Content-Type": [
        "application/xml"
    ],
    "Date": [
        "Mon, 11 Feb 2019 23:00:33 GMT"
    ],
    "Server": [
        "AmazonS3"
    ],
    "X-Amz-Id-2": [
        "v208XITkl8dU+ePcHuO/upxrpJPiyUGInjO9c03CmLKHTOZFy/BYVQa/Vjm1vT2X0DckbJqehdk="
    ],
    "X-Amz-Request-Id": [
        "28B4106E0A4F3417"
    ]
}

END RequestId: 9ff22b3d-3d1e-4f37-b98e-9e3ab53fe9e1
REPORT RequestId: 9ff22b3d-3d1e-4f37-b98e-9e3ab53fe9e1	Duration: 154.45 ms	Billed Duration: 200 ms Memory Size: 1024 MB	Max Memory Used: 26 MB

lambda.go

package main

import (
  "context"
  "database/sql"
  "encoding/json"
  "fmt"
  "github.com/kontrollanten/aws-lambda-go/cfn"
  "github.com/aws/aws-lambda-go/lambda"
  _ "github.com/go-sql-driver/mysql"
  "os"
)

func handler(ctx context.Context, event cfn.Event) (rid string, data map[string]interface{}, err error) {
  jsonEvent, e := json.Marshal(event)
  rid = "DB_MIGRATION"
  if e != nil {
    fmt.Printf("json Marshal of event failed %v", e)
    jsonEvent = []byte{}
  }

  fmt.Printf("Starting with\ncontext %v\njsonEvent %v\n", ctx, string(jsonEvent[:]))

  if event.RequestType != cfn.RequestCreate {
    fmt.Println("Request type is not of type create, returning")
    fmt.Printf("physicalResourceID: %s\n", rid)
    return
  }
  fmt.Printf("physicalResourceID: %s\n", rid)

  fmt.Printf("Connection to database at %s.\n", os.Getenv("DB_ADDRESS"))
  db, err := sql.Open("mysql", fmt.Sprintf("%s:%s@(%s)/", os.Getenv("DB_MASTER_USER"), os.Getenv("DB_MASTER_PASSWORD"), os.Getenv("DB_ADDRESS")))
  if err != nil {
    panic(err)
  }
  defer db.Close()

  fmt.Printf("Database connection open to %s.\n", os.Getenv("DB_ADDRESS"))

  defer func() {
    if r := recover(); r != nil {
      err = fmt.Errorf("handler: Failed to migrate DB: %v", r)
    }
  }()

  MigrateDb(db)

  return
}

func main() {
  lambda.Start(cfn.LambdaWrap(handler))
}

serverless.yml

service: update-shelters

provider:
  name: aws
  runtime: go1.x
  tags:
    project: hitta-skyddsrum

  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "rds:CreateDBInstance"
        - "rds:DeleteDBInstance"
      Resource: { "Fn::Join" : ["", ["arn:aws:rds:::", { "Ref" : "SheltersDB" } ] ]  }
    - Effect: "Allow"
      Action:
        - "s3:GetObject"
        - "s3:PutObject"
      Resource: "arn:aws:s3:::${env:BUCKET_NAME}/*"
    - Effect: "Allow"
      Action:
        - "s3:ListBucket"
      Resource: "arn:aws:s3:::${env:BUCKET_NAME}"

package:
 exclude:
   - ./**
 include:
   - ./lambda-bin/**

functions:
  dbMigration:
    handler: lambda-bin/migrate-db
    environment:
      DB_MASTER_USER: ${env:DB_MASTER_USER}
      DB_MASTER_PASSWORD: ${env:DB_MASTER_PASSWORD}
      DB_ADDRESS:
        "Fn::GetAtt": [ SheltersDB, Endpoint.Address ]
    vpc:
      securityGroupIds:
        - Ref: SheltersVPCSecurityGroup
      subnetIds:
        - Ref: SheltersSubnet1
        - Ref: SheltersSubnet2
  shapeToCsv:
    handler: lambda-bin/shape-to-csv
  importCsvToMysql:
    handler: lambda-bin/import-csv-to-mysql
    timeout: 300
    environment:
      DB_MASTER_USER: ${env:DB_MASTER_USER}
      DB_MASTER_PASSWORD: ${env:DB_MASTER_PASSWORD}
      DB_ADDRESS:
        "Fn::GetAtt": [ SheltersDB, Endpoint.Address ]
    vpc:
      securityGroupIds:
        - Ref: SheltersVPCSecurityGroup
      subnetIds:
        - Ref: SheltersSubnet1
        - Ref: SheltersSubnet2

resources:
  Resources:
    S3BucketSheltersBucket:
      Type: AWS::S3::Bucket
      DependsOn:
        - ShapeToCsvPermission
        - ImportCsvToMysqlPermission
      Properties:
        BucketName: ${env:BUCKET_NAME}
        NotificationConfiguration:
          LambdaConfigurations:
            - Event: "s3:ObjectCreated:*"
              Function: !GetAtt
                - ShapeToCsvLambdaFunction
                - Arn
              Filter:
                S3Key:
                  Rules:
                    -
                      Name: suffix
                      Value: zip
            - Event: "s3:ObjectCreated:*"
              Function: !GetAtt
                - ImportCsvToMysqlLambdaFunction
                - Arn
              Filter:
                S3Key:
                  Rules:
                    -
                      Name: suffix
                      Value: csv
    ImportCsvToMysqlPermission:
      DependsOn:
        - ImportCsvToMysqlLambdaFunction
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !GetAtt
          - ImportCsvToMysqlLambdaFunction
          - Arn
        Action: "lambda:InvokeFunction"
        Principal: "s3.amazonaws.com"
        SourceArn: "arn:aws:s3:::${env:BUCKET_NAME}"
    ShapeToCsvPermission:
      DependsOn:
        - ImportCsvToMysqlLambdaFunction
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !GetAtt
          - ShapeToCsvLambdaFunction
          - Arn
        Action: "lambda:InvokeFunction"
        Principal: "s3.amazonaws.com"
        SourceArn: "arn:aws:s3:::${env:BUCKET_NAME}"
    SheltersDB:
      Type: AWS::RDS::DBInstance
      Properties:
        AllocatedStorage: 5
        DBInstanceClass: db.t2.micro
        DBSubnetGroupName: !Ref SheltersDBSubnetGroup
        VPCSecurityGroups:
          - Fn::GetAtt: [ SheltersVPCSecurityGroup, GroupId ]
        Engine: MySQL
        MasterUsername: ${env:DB_MASTER_USER}
        MasterUserPassword: ${env:DB_MASTER_PASSWORD}
    SheltersDBMigrationTrigger:
      Type: Custom::DBMigration
      DependsOn:
        - SheltersDB
      Properties:
        ServiceToken: !GetAtt
          - DbMigrationLambdaFunction
          - Arn
    SheltersDBSubnetGroup:
      Type: AWS::RDS::DBSubnetGroup
      Properties:
        DBSubnetGroupDescription: "Subnet for DB connections"
        SubnetIds:
          - Ref: SheltersSubnet1
          - Ref: SheltersSubnet2
    SheltersSubnet1:
      Type: AWS::EC2::Subnet
      Properties:
        AvailabilityZone: !Select [ 0, {Fn::GetAZs: ""} ]
        CidrBlock: 10.0.1.0/24
        VpcId: !Ref SheltersVPC
    SheltersSubnet2:
      Type: AWS::EC2::Subnet
      Properties:
        AvailabilityZone: !Select [ 1, {Fn::GetAZs: ""} ]
        CidrBlock: 10.0.2.0/24
        VpcId: !Ref SheltersVPC
    SheltersVPCSecurityGroup:
      Type: AWS::EC2::SecurityGroup
      Properties:
        GroupDescription: "Security group for DB connections"
        VpcId: !Ref SheltersVPC
    SheltersVPCSecurityGroupIngress:
      Type: AWS::EC2::SecurityGroupIngress
      Properties:
        GroupId: !Ref SheltersVPCSecurityGroup
        IpProtocol: tcp
        FromPort: "3306"
        ToPort: "3306"
        SourceSecurityGroupId: !Ref SheltersVPCSecurityGroup
    SheltersVPC:
      Type: AWS::EC2::VPC
      Properties:
        CidrBlock: 10.0.0.0/16
    SheltersRouteTable:
      Type: AWS::EC2::RouteTable
      Properties:
        VpcId: !Ref SheltersVPC
    SheltersSubnet1Association:
      Type: AWS::EC2::SubnetRouteTableAssociation
      Properties:
        SubnetId: !Ref SheltersSubnet1
        RouteTableId: !Ref SheltersRouteTable
    SheltersSubnet2Association:
      Type: AWS::EC2::SubnetRouteTableAssociation
      Properties:
        SubnetId: !Ref SheltersSubnet2
        RouteTableId: !Ref SheltersRouteTable
    SheltersVPCS3Endpoint:
      Type: AWS::EC2::VPCEndpoint
      Properties:
        VpcId: !Ref SheltersVPC
        PolicyDocument: "{\"Version\":\"2008-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"*\",\"Resource\":\"*\"}]}"
        RouteTableIds:
          - !Ref SheltersRouteTable
        ServiceName: !Join ['', ['com.amazonaws.', !Ref 'AWS::Region', '.s3']]