Wagtail on AWS Lambda, with Zappa

Aim

A demonstration of Wagtail running on AWS Lambda + API Gateway using Zappa for deployment.

This is not a Production solution, it is an insecure setup focusing solely on getting Wagtail to run on Lambda in the simplest way possible (and at zero cost if it is not used beyond this demonstration (or close to zero depending on how much you exercise the S3 bucket)).

caveat emptor!

Stack

Wagtail (latest, python3) running on Lambda
Sqlite database (using s3sqlite https://blog.zappa.io/posts/s3sqlite-a-serverless-relational-database)
S3 bucket for Zappa deployment, Sqlite database hosting and static asset serving
Zappa for deployment (with its Django utils)

An option for using RDS instead of Sqlite is given at the end.

Pre-requisites

Python 3 on the machine where this set up is taking place

Wagtail 2.x and above only runs on Python 3 so this is necessary for running it locally

AWS account

https://portal.aws.amazon.com/billing/signup

Credentials for an IAM User in the AWS account

These credentials need to be available on the machine where this set up is taking place, typically as a profile in ~/.aws/config and ~/.aws/credentials files.

See the Configuration Settings and Precedence section of

https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html

for ways of making the credentials available (either using the aws cli tool of manually).

Permissions for the IAM User

The exact permissions required for all the actions Zappa needs to perform are hard to determine (e.g: Miserlou/Zappa#244 and Miserlou/Zappa#849).

A completely permissive, and therefore insecure, approach is for the IAM User to have the AdministratorAccess permission.

IAM Users can be created and permissions granted at

https://console.aws.amazon.com/iam/home

The IAM User name should match the name of a profile in the credentials in the previous step (the main AWS account credentials can be used or specific credentials created for this IAM User).

Set up

In this demonstration we will use zagtail as a name for the set up, you can use whatever you like.

Install packages

Create a virtual environment for installing pip packages:

python3 -m venv ~/.venvs/zagtail-env
source ~/.venvs/zagtail-env/bin/activate

(or the path to wherever you keep virtual environments)

Install Wagtail and Zappa:

pip install wagtail zappa zappa-django-utils
pip install --upgrade pip  # upgrade beyond 10.x, the current 18.x is okay

Go to a directory where you want to create the empty Wagtail site locally and create it:

wagtail start zagtail
cd zagtail

Initialise Zappa

It will prompt for configuration:

Choose the dev environment
Choose the profile matching you permissive IAM User
Let it create an S3 bucket for you (it will be prefixed zappa-)
Set the path to the Django setting module to be zagtail.settings.dev (don't miss the .dev)
Choose 'n' for deploying globally

zappa init

This will generate a zappa_settings.json file, e.g.

{
    "dev": {
        "django_settings": "zagtail.settings.dev",
        "profile_name": "zappa-zagtail",
        "project_name": "zagtail",
        "runtime": "python3.6",
        "s3_bucket": "zappa-ge9mqpo3s"
    }
}

Your profile_name and s3_bucket will be different.

Include zappa_django_utils in your INSTALLED_APPS and add Zappa's SQLite backend by editing zagtail/settings/dev.py to add the following to the end of it (replacing the BUCKET name with the one from your zappa_settings.json:

INSTALLED_APPS += ('zappa_django_utils',)

DATABASES = {
    'default': {
        'ENGINE': 'zappa_django_utils.db.backends.s3sqlite',
        'NAME': 'zagtail-sqlite.db',
        'BUCKET': 'zappa-ge9mqpo3s'
    }
}

(Note that the Sqlite database will live in the publicly accessible S3 bucket and, since the local runserver will also use these settings (its wsgi.py uses dev.py), both the remote and local Wagtail sites will share the same database! Zappa, by default, writes a copy to /tmp/zagtail-sqlite.db and synchronises to the S3 bucket file between requests.)

Deploy the site using Zappa

Zappa will create its own IAM Role and add permissions to it, create a stack necessary to run on Lambda, upload the site and create an API Gateway to access it:

zappa deploy dev

but, although the set up will have worked Zappa will report:

Error: Warning! Status check on the deployed lambda failed. A GET request to '/' yielded a 500 response code.

because Wagtail itself cannot start correctly from an unmigrated database (in fact as it's a Sqlite database it doesn't exist at all in the S3 bucket yet).

If the application deployed was in a runnable state (returning a 200 code) Zappa would report:

Deployment complete!: https://clu3eolho5.execute-api.eu-west-1.amazonaws.com/dev

Your domain will be different. This is what you will see on subsequent successful deployments once Wagtail is in a runnable state.

Until then you can get the API Gateway URL from:

zappa status dev

Note that the site is rooted at /dev, the Zappa environment we're using here. If you visit the URL you'll see the 500 error message from Django:

OperationalError at /
no such table: wagtailcore_site

Note, if you got a 502 error at the end of zappa deploy dev that is indicative of your IAM User not having enough permissions for Zappa to set up the infrastructure.

To get Wagtail running, migrate the database (which will then create the Sqlite database file in the S3 bucket):

zappa manage dev migrate

We can also create a Django/Wagtail Admin user:

zappa manage dev create_admin_user

Note, this is Zappa's create_admin_user command, not Django auth's createsuperuser command. The former automatically creates an admin username and gives you a password. Using this gets around the issue of the needing input for createsuperuser when Lambda has no command line.

If you visit the site now you will see the unstyled Wagtail front end and if you visit /dev/admin you'll see an unstyled Wagtail Admin (you'll be able to log in using the admin user and the password Zappa's create_admin_user gave you).

Configure static assets

To set up Django to serve the static assets from the S3 bucket in order to get a styled site and Wagtail Admin:

pip install django-storages boto3

Add the following configuration to settings/zagtail/dev.py (again replacing the bucket name with your one:

INSTALLED_APPS += ('storages',)
AWS_STORAGE_BUCKET_NAME = 'zappa-ge9mqpo3s'
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

Setting AWS_S3_CUSTOM_DOMAIN isn't essential, by default django-storages will use AWS_STORAGE_BUCKET_NAME.s3.amazonaws.com anyway in Django's {% static %} template tag. It could, for example, instead be AWS_STORAGE_BUCKET_NAME.eu-west-1.s3.amazonaws.com which points to the same bucket but includes the AWS region in the bucket's URL (where eu-west-1 was the region for the bucket in this case).

Note, we're also insecurely setting the media storage to this bucket as well. Images uploaded in Wagtail will go here.

(Note also that AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY aren't set. So how is Django Storages allowed to write to the bucket, you may ask? When they are not set Django Storages looks for them in environment variables. When the Zappa attaches its role with appropriate policies (zagtail-dev-ZappaLambdaExecutionRole) to the Lambda Function the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables are automatically created by Lambda. The ID starts with ASIA indicating it's an IAM Role credential rather than an IAM User credential, which starts with AKIA. These environment variables are not visible in the AWS Lambda Function Console but can be seen from Zappa, e.g. zappa invoke dev "print(os.environ.get('AWS_ACCESS_KEY_ID'))" --raw. The net effect is that you don't set credentials yourself.)

Collect the static files in the bucket. Note we are now using the update command to deploy changes. The deploy command is only used initially to create the stack. The manage command is then used to run Django's collectstatic management command:

zappa update dev
zappa manage dev "collectstatic --noinput"

Fonts won't work until a cross-origin permissive CORS policy is set on the bucket. Go to your S3 bucket properties in the AWS console, and under "Permissions", click on "Add CORS Configuration", and replace the contents with:

<CORSConfiguration>
  <CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>HEAD</AllowedMethod>
  </CORSRule>
</CORSConfiguration>

Visiting the front end and Wagtail Admin should now show Wagtail running on AWS Lambda in all its glory!

RDS

Instead of using Sqlite as the database you can use RDS to run Wagtail on MySQL, PostgreSQL or Aurora. This example changes from Sqlite to PostgreSQL.

Note that unless your AWS Account is less than 12 months old you will be charged for the amount of time your RDS database is running. The AWS Free Tier is available for 12 months after the Account was created.

Go to RDS in the AWS Console and click on Create Database. On the Select Engine page tick "Only enable options eligible for RDS Free Usage Tier", select PostgreSQL then Next.

In Settings set "DB instance identifier" to zagtail-rds and "Master username" to zagtail_rds_user and set a password, then Next.

Under "Configure advanced settings" set "Database name" to zagtail_db leaving everything else as it is (including "Public accessibility" set to Yes). Then click on Create Database and wait a few minutes for it to be created.

In the meantime view the database details and under "Security groups" click the launch wizard link. Under "Inbound" edit the Source to "Anywhere" - it will have been automatically set to the IP address that you are using to access the AWS Console ("Anywhere" changes it to 0.0.0.0/0). Without changing this the Wagtail Lambda Function will not be able to access the database and web requests / management commands will time out.

Once the database has been created the database details page, under "Connect" will show the Endpoint for the RDS. This is the Host parameter for Django's database configuration.

With the empty database set up we can configure Django to use it.

Install PostgreSQL psychopg2 database driver in the virtual environment:

pip install psycopg2

Then replace the Sqlite database configuration in zagtail/settings/dev.py with the PostgreSQL configuration, replacing the PASSWORD and HOST (Endpoint) values with your own ones:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'zagtail_db',
        'USER': 'zagtail_rds_user',
        'PASSWORD': 'YOUR_PASSWORD',
        'HOST': 'zagtail_db-rds.cpexs1ktzihq.eu-west-1.rds.amazonaws.com',
        'PORT': '5432',
    }
}

Then apply this change, migrate the database and create an admin user, as you did for the Sqlite database:

zappa update dev
zappa manage dev migrate
zappa manage dev create_admin_user

Wagtail will then be running off the PostgreSQL database.

Tear down

You can remove the demonstration site from AWS by running:

zappa undeploy dev

In the AWS console go to S3 to delete the bucket

https://console.aws.amazon.com/s3/home

and go to IAM then Roles to delete the Zappa role zagtail-dev-ZappaLambdaExecutionRole.

https://console.aws.amazon.com/iam/home#/roles

If you created the RDS database, go to RDS, select the database and click on Modify. Then deselect "Enable deletion protection". After applying this, select the database again, click on Actions and Delete.

https://console.aws.amazon.com/rds/home#databases:

Production stack checklist

As noted the above stack is for demonstration only and is insecure. A production stack would include a number of other parts that will be documented in another example.

Such a stack would include considerations for:

Locked down IAM User permissions
Separate buckets for Zappa and Static assets/Media files
Non-Sqlite database on AWS RDS (PostgreSQL, MySQL or Aurora)
Virtual Private Cloud for connecting services
Environment variables for Django settings
Custom domain setup (for Wagtail and static assets) via Route S3
Locked down CORS policy
AWS Elasticache for Django caching
Front-end caching
Auto-scaling

nealtodd/wagtail-on-zappa.md