Django migrations without downtime

migrations.md

Django Migrations without Downtime

The following instructions describe a set of processes allowing you to run Django database migrations against a production database without having to bring the web service down.

Note that in the below instructions, migrations are all run manually at explicit points, and are not an automatic part of the deployment process.

Adding Fields or Tables

Adding a (nullable) field or a new table

1. Make the model or column addition in your code.
2. Generate the model-change migrations locally.
3. Make two separate pull requests, one containing the model change, one containing the newly-generated migrations. Note that the model-change migration should only contain the model change code, not any supplemental code using the newly-structured model.
4. Merge and deploy the migration pull request.
5. Run the new migration on the production database. Additional fields in the production database will not cause any problems if the production model code is not expecting them to be there.
6. Merge the model-change pull request.
7. Deploy the code with the model change. It will now seamlessly start reading from the previously created column/table.

Adding a NOT NULL field

In order to add a NOT NULL field, you should first add a nullable field with the steps above, then follow the following additional steps:

1. Make the model change to null=False. Also ensure your code never inserts or updates None values into the field you are converting to NOT NULL.
2. Generate the migration for this change.
3. Make two separate pull requests for these changes: one for the models, one for the migration.
4. Merge and deploy the code change pull request.
5. Merge the migration pull request.
6. Run the migration on the production database.

Notes on adding NOT NULL columns in very large tables

Django migrations will insist on a default value when making a NOT NULL field. This value is used to populate empty columns during the migration. This update process will lock the postgres table for writes until it has completed. For relatively small tables (<100,000 rows), this is probably fine. For exceptionally large tables though, this could be a problem, because the locked tables could prevent requests from executing while the migration completes. For this reason, consider making new fields nullable in especially large tables (and to a lesser extent in all tables). This may have a cost to your application code, but is probably optimal in comparison to failing to respond to requests.

Removing Fields or Tables

Removing a nullable field or a table

Essentially, this is the same as adding, but with the steps in a slightly different order.

1. Remove all usages of the model or column to be deleted from your application code.
2. Deploy a version of the code which still has the model/column to be deleted present, but not used by any of the code.
3. Make the model or column removal in your application code.
4. Generate the removal migration.
5. Merge the model change pull request.
6. Deploy the code with the model changes. Ensure there are no errors resulting from lingering references to what is being deleted.
7. Merge the migration pull request.
8. Deploy the code with the migration.
9. Explicitly run the migration on the production database.

Removing a NOT NULL field

In order to remove a NOT NULL field, you first need to migrate it to a nullable field, then follow the steps above to remove a nullable field. The steps to convert a NOT NULL field to a nullable field are:

1. Set your field's model to null=True
2. Generate your migration for this change.
3. Make two separate pull requests: one with the model-change, one with the migration change.
4. Merge and deploy the migration pull request.
5. Run the migration on the production database.
6. Merge and deploy your model change code.

After following these steps, follow the steps above to remove the nullable field.

Good post! I'd just add a note that if the table is too large (my case almost 1B rows) the migration with adding a null=True field can still cause downtime. In this case it's nice to add this to the migration:

https://docs.djangoproject.com/en/3.1/howto/writing-migrations/#non-atomic-migrations

What is the reason for running manually? What would prevent the migrations to be run automatically in a separate process?

@stefaneg I was wondering the same thing. I think it's just that, at that point, you don't have to make a deployment since there's no change in code. You can just run the migration in production and proceed to the next step. Personally, I prefer to make the deployment anyway.