Lambda Python Environment SSM Doc

lambda_python_env.json

{
  "schemaVersion":"0.3",
  "description":"Takes a Lambda AMI and sets up Python3 + virtualenv",
  "assumeRole":"{{AutomationAssumeRole}}",
  "parameters":{
    "SourceAmiId":{
      "type":"String",
      "description":"(Required) The source Lambda compliant AMI ID.",
      "default":"ami-f6035893"
    },
    "PythonVersion":{
      "type":"String",
      "description":"(Required) The Python 3 version to use.",
      "default":"3.6.1"
    },
    "OsUser":{
      "type":"String",
      "description":"(Required) The ID of the non-root user.",
      "default":"ec2-user"
    },
    "InstanceIamRole":{
      "type":"String",
      "description":"(Required) The name of the role that enables Systems Manager (SSM) to manage the instance.",
      "default": "ManagedInstanceProfile"
    },
    "AutomationAssumeRole":{
      "type":"String",
      "description":"(Required) The ARN of the role that allows Automation to perform the actions on your behalf.",
      "default":"arn:aws:iam::{{global:ACCOUNT_ID}}:role/AutomationServiceRole"
    },
    "TargetAmiName":{
      "type":"String",
      "description":"(Optional) The name of the new AMI that will be created. Default is a system-generated string including the source AMI id, and the creation time and date.",
      "default":"PyLambdaDeployAMI-{{global:DATE_TIME}}"
    },
    "InstanceType":{
      "type":"String",
      "description":"(Optional) Type of instance to launch as the workspace host. Instance types vary by region. Default is t2.micro.",
      "default":"t2.micro"
    }
  },
  "mainSteps":[
    {
      "name":"launchInstance",
      "action":"aws:runInstances",
      "maxAttempts":3,
      "timeoutSeconds":300,
      "onFailure":"Abort",
      "inputs":{
        "ImageId":"{{SourceAmiId}}",
        "InstanceType":"{{InstanceType}}",
        "UserData":"IyEvYmluL2Jhc2gNCg0KZnVuY3Rpb24gZ2V0X2NvbnRlbnRzKCkgew0KICAgIGlmIFsgLXggIiQod2hpY2ggY3VybCkiIF07IHRoZW4NCiAgICAgICAgY3VybCAtcyAtZiAiJDEiDQogICAgZWxpZiBbIC14ICIkKHdoaWNoIHdnZXQpIiBdOyB0aGVuDQogICAgICAgIHdnZXQgIiQxIiAtTyAtDQogICAgZWxzZQ0KICAgICAgICBkaWUgIk5vIGRvd25sb2FkIHV0aWxpdHkgKGN1cmwsIHdnZXQpIg0KICAgIGZpDQp9DQoNCnJlYWRvbmx5IElERU5USVRZX1VSTD0iaHR0cDovLzE2OS4yNTQuMTY5LjI1NC8yMDE2LTA2LTMwL2R5bmFtaWMvaW5zdGFuY2UtaWRlbnRpdHkvZG9jdW1lbnQvIg0KcmVhZG9ubHkgVFJVRV9SRUdJT049JChnZXRfY29udGVudHMgIiRJREVOVElUWV9VUkwiIHwgYXdrIC1GXCIgJy9yZWdpb24vIHsgcHJpbnQgJDQgfScpDQpyZWFkb25seSBERUZBVUxUX1JFR0lPTj0idXMtZWFzdC0xIg0KcmVhZG9ubHkgUkVHSU9OPSIke1RSVUVfUkVHSU9OOi0kREVGQVVMVF9SRUdJT059Ig0KDQpyZWFkb25seSBTQ1JJUFRfTkFNRT0iYXdzLWluc3RhbGwtc3NtLWFnZW50Ig0KIFNDUklQVF9VUkw9Imh0dHBzOi8vYXdzLXNzbS1kb3dubG9hZHMtJFJFR0lPTi5zMy5hbWF6b25hd3MuY29tL3NjcmlwdHMvJFNDUklQVF9OQU1FIg0KDQppZiBbICIkUkVHSU9OIiA9ICJjbi1ub3J0aC0xIiBdOyB0aGVuDQogIFNDUklQVF9VUkw9Imh0dHBzOi8vYXdzLXNzbS1kb3dubG9hZHMtJFJFR0lPTi5zMy5jbi1ub3J0aC0xLmFtYXpvbmF3cy5jb20uY24vc2NyaXB0cy8kU0NSSVBUX05BTUUiDQpmaQ0KDQppZiBbICIkUkVHSU9OIiA9ICJ1cy1nb3Ytd2VzdC0xIiBdOyB0aGVuDQogIFNDUklQVF9VUkw9Imh0dHBzOi8vYXdzLXNzbS1kb3dubG9hZHMtJFJFR0lPTi5zMy11cy1nb3Ytd2VzdC0xLmFtYXpvbmF3cy5jb20vc2NyaXB0cy8kU0NSSVBUX05BTUUiDQpmaQ0KDQpjZCAvdG1wDQpGSUxFX1NJWkU9MA0KTUFYX1JFVFJZX0NPVU5UPTMNClJFVFJZX0NPVU5UPTANCg0Kd2hpbGUgWyAkUkVUUllfQ09VTlQgLWx0ICRNQVhfUkVUUllfQ09VTlQgXSA7IGRvDQogIGVjaG8gQVdTLVVwZGF0ZUxpbnV4QW1pOiBEb3dubG9hZGluZyBzY3JpcHQgZnJvbSAkU0NSSVBUX1VSTA0KICBnZXRfY29udGVudHMgIiRTQ1JJUFRfVVJMIiA+ICIkU0NSSVBUX05BTUUiDQogIEZJTEVfU0laRT0kKGR1IC1rIC90bXAvJFNDUklQVF9OQU1FIHwgY3V0IC1mMSkNCiAgZWNobyBBV1MtVXBkYXRlTGludXhBbWk6IEZpbmlzaGVkIGRvd25sb2FkaW5nIHNjcmlwdCwgc2l6ZTogJEZJTEVfU0laRQ0KICBpZiBbICRGSUxFX1NJWkUgLWd0IDAgXTsgdGhlbg0KICAgIGJyZWFrDQogIGVsc2UNCiAgICBpZiBbWyAkUkVUUllfQ09VTlQgLWx0IE1BWF9SRVRSWV9DT1VOVCBdXTsgdGhlbg0KICAgICAgUkVUUllfQ09VTlQ9JCgoUkVUUllfQ09VTlQrMSkpOw0KICAgICAgZWNobyBBV1MtVXBkYXRlTGludXhBbWk6IEZpbGVTaXplIGlzIDAsIHJldHJ5Q291bnQ6ICRSRVRSWV9DT1VOVA0KICAgIGZpDQogIGZpIA0KZG9uZQ0KDQppZiBbICRGSUxFX1NJWkUgLWd0IDAgXTsgdGhlbg0KICBjaG1vZCAreCAiJFNDUklQVF9OQU1FIg0KICBlY2hvIEFXUy1VcGRhdGVMaW51eEFtaTogUnVubmluZyBVcGRhdGVTU01BZ2VudCBzY3JpcHQgbm93IC4uLi4NCiAgLi8iJFNDUklQVF9OQU1FIiAtLXJlZ2lvbiAiJFJFR0lPTiINCmVsc2UNCiAgZWNobyBBV1MtVXBkYXRlTGludXhBbWk6IFVuYWJsZSB0byBkb3dubG9hZCBzY3JpcHQsIHF1aXR0aW5nIC4uLi4NCmZp",
        "MinInstanceCount":1,
        "MaxInstanceCount":1,
        "IamInstanceProfileName":"{{InstanceIamRole}}"
      }
    },
    {
      "name":"updateOSSoftware",
      "action":"aws:runCommand",
      "maxAttempts":3,
      "timeoutSeconds":3600,
      "onFailure":"Abort",
      "inputs":{
        "DocumentName":"AWS-RunShellScript",
        "InstanceIds":[
          "{{launchInstance.InstanceIds}}"
        ],
        "Parameters":{
          "commands":[
            "set -e",
            "yum update -y",
            "yum install -y gcc zlib zlib-devel openssl openssl-devel"
          ]
        }
      }
    },
    {
      "name":"setupPython",
      "action":"aws:runCommand",
      "maxAttempts":3,
      "timeoutSeconds":3600,
      "onFailure":"Abort",
      "inputs":{
        "DocumentName":"AWS-RunShellScript",
        "InstanceIds":[
          "{{launchInstance.InstanceIds}}"
        ],
        "Parameters":{
          "commands":[
            "set -e",
            "cd /tmp",
            "wget https://www.python.org/ftp/python/{{PythonVersion}}/Python-{{PythonVersion}}.tgz",
            "tar xzvf Python-{{PythonVersion}}.tgz",
            "cd Python-{{PythonVersion}} && ./configure && make",
            "make install",
            "/usr/local/bin/pip3 install --upgrade pip",
            "/usr/local/bin/pip3 install virtualenv"
          ]
        }
      }
    },
    {
      "name":"setupUserEnvironment",
      "action":"aws:runCommand",
      "maxAttempts":3,
      "timeoutSeconds":3600,
      "onFailure":"Abort",
      "inputs":{
        "DocumentName":"AWS-RunShellScript",
        "InstanceIds":[
          "{{launchInstance.InstanceIds}}"
        ],
        "Parameters":{
          "commands":[
            "set -e",
            "su {{OsUser}} -c '/usr/local/bin/virtualenv ~/venv-python{{PythonVersion}}'",
            "su {{OsUser}} -c 'aws configure set default.region {{global:REGION}}'"
          ]
        }
      }
    },
    {
      "name":"stopInstance",
      "action":"aws:changeInstanceState",
      "maxAttempts":3,
      "timeoutSeconds":1200,
      "onFailure":"Abort",
      "inputs":{
        "InstanceIds":[
          "{{launchInstance.InstanceIds}}"
        ],
        "DesiredState":"stopped"
      }
    },
    {
      "name":"createImage",
      "action":"aws:createImage",
      "maxAttempts":3,
      "onFailure":"Abort",
      "inputs":{
        "InstanceId":"{{launchInstance.InstanceIds}}",
        "ImageName":"{{TargetAmiName}}",
        "NoReboot":true,
        "ImageDescription":"AMI Generated by EC2 Automation on {{global:DATE_TIME}} from {{SourceAmiId}}"
      }
    },
    {
      "name":"terminateInstance",
      "action":"aws:changeInstanceState",
      "maxAttempts":3,
      "onFailure":"Continue",
      "inputs":{
        "InstanceIds":[
          "{{launchInstance.InstanceIds}}"
        ],
        "DesiredState":"terminated"
      }
    }
  ],
  "outputs":[
    "createImage.ImageId"
  ]
}

This is an EC2 Systems Manager Automation Document for setting up a Python 3.x environment for Lambda functions which use native code python modules. It does the following:

• Updates the system
• Installs a few system packages that are needed to compile python
• Downloads python from the official repo and installs it to /usr/local
• Upgrades pip
• Installs virtualenv
• Sets the default AWS CLI region to that of the region where the document is being run for the ec2-user
• Sets up a python 3 virtual environment in ~/venv-python{{PythonVersion}} for the ec2-user

Warnings

• You are charged for the resources booted, unless you're on the Free Tier for EC2 and are using t2.micro instances
• Until a VPC endpoint is released for SSM, the agent needs to communicate with the SSM service through public IP. This means it will be counted against the EC2 -> EC2 via public IP pricing
• If the automation fails it will not cleanup the EC2 instance. You will need to terminate instances manually in that case

To setup the automation doc:

1. Login to the AWS console if you're not already
2. Visit the EC2 services page
3. Under "SYSTEMS MANAGER SHARED RESOURCES" click "Documents"
4. Click "Create Document"
5. Enter a name for the document
6. For "Document Type" select "Automation"
7. Replace the text box content with the above JSON file
8. Open up a new tab
9. Visit the Supported Lambda Environment page
10. Copy the AMI name
11. In the new tab, pull up the EC2 console (Make sure you are in the proper region where you will be running the Automation document!)
12. Click on "AMIs"
13. Change the "Owned By Me" filter to "Public Image"
14. Paste in the AMI name you copied from step 10 into the search bar and press enter
15. This will give you the correct AMI ID for the specific region, which you will copy the value of
16. Go back to the tab with the Systems Manager JSON document
17. Replace the "default":"ami-f6035893" (or similar if this document has been updated with a new ID) with "default":"[ami ID from step 15]"
18. Click on "Create Document"
19. If you haven't already, run the Cloud Formation Template for the SSM Automation IAM roles. The region defaults to us-east-1 so be sure to switch the console to the appropriate region if different.
20. Go back to the EC2 console for the region that has the AMI from step 15
21. Under "SYSTEMS MANAGER SERVICES" click "Automations"
22. Click "Run Automation"
23. In the list of documents click the name of the document you just created
24. Leave the version as is
25. Input Parameters contains the list of options to pass to the automation. Adjust what you like here. If you forgot to set the default AMI ID for the lambda AMI you can change it here.
26. When everything looks good click "Run Automation"

This will boot up an EC2 instance, setup the environment, stop the instance, and then create an AMI. Now in the "AMIs" part of the EC2 console you should see PyLambdaDeployAMI-(some date here) or whatever you named your AMI in the Automation parameters. You can now use this to boot an EC2 instance which contains the python virtual environment.

From there when you login, you can simply cd into ~/venv-python{{PythonVersion}} and run source bin/activate. Now install whatever python modules are needed using pip. When that's done:

cd $VIRTUAL_ENV/lib/python3.6/site-packages
zip -r9 ~/MyLambda.zip *
cd ~/your/lambda/handler/dir
zip -g MyLambda.zip lambda.py

This will give you a ZIP file that you can then upload to S3 or through the lambda API directly. It's a good idea to do this using the AWS CLI, using a role that has permissions to interact with the Lambda/S3 API.