Database rollback options in Django

README.md

Batching migrations

The idea here is that if you know which migrations were in version 2.0.3 of your project and which were in version 2.0.4 then setA - setB gives you the list of migrations you need to undo.

Django migrations give you a directed acyclic graph which describes how to get from the current database state to the target state. But there is no mechanism by which you can perform tasks like revert all the migrations that just ran in the last deployment.

Here is a quick recipe for batching Django migrations to allow you to do things like that.

Before you tag your project you do:

django freeze 2.0.3

This generates a json file in which looks like this:

{
  "app-a": "0001_initial",
  "app-b": "0002_add_a_field"
}

It lists all the apps configured in your project and their most recent migration. It's stored in your project folder in a releases subdirectory. It should be added to your VCS.

With this process in place we now have a list of targets for each tag. To go from any future database state back to 2.0.3 we simply have to undo any migration that is not reference in the json file (or one of its dependencies).

And we have a command to do that:

django rollback 2.0.3

Caveats

When merges are involved, Django doesn't always revert migrations cleanly

If in 2.0.3 you had migrations that looked like this:

A -> B

In this example you could unapply B with:

django migrate my_app A

Now you deploy 2.0.4 and that graph becomes:

A - > B - > D
 \- > C -/

(Where D is a merge migration that depends on B and C).

Now in 2.0.3 our metadata says that B is the latest migration. So under the hood our migration engine is going to try something like:

django migrate my_app B

You might hope that C and D are reverted, but only D is reverted.

It may be possible to generate a list of all applied migrations and a list of all migrations the B depends on and work out that C and D should be reverted, then use Django's migration planner to execute them in a safe order. This has not been tested yet.

Reverting fails a lot

It's quite common to change the database level constraints - for example to suddenly allow NULL data. You'll almost certainly end up with data that doesn't satisfy this constraint. So any attempt to revert will fail. Similar problems existing when changing the size of a field.

So relying on this as a strategy for backing our changes is risky! More testing should be added.

freeze.py

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

import os
import json

from django.conf import settings
from django.core.management.base import BaseCommand, CommandError
from django.db import DEFAULT_DB_ALIAS, connections
from django.db.migrations.loader import MigrationLoader


class Command(BaseCommand):

    help = "Shows all available migrations for the current project"

    def add_arguments(self, parser):
        parser.add_argument(
            'release',
            action='store',
            help='The release file to generate'
        )
        parser.add_argument(
            '--database',
            action='store',
            dest='database',
            default=DEFAULT_DB_ALIAS,
            help='Nominates a database to synchronize. Defaults to the "default" database.'
        )

    def handle(self, *args, **options):
        releases_dir = getattr(
            settings,
            "RELEASES_DIR",
            os.path.join(settings.PROJECT_PATH, "releases"),
        )

        db = options.get('database')

        loader = MigrationLoader(connections[db], ignore_no_migrations=True)
        graph = loader.graph
        app_names = sorted(loader.migrated_apps)

        leaf_migrations = {}
        for app_name in app_names:
            nodes = graph.leaf_nodes(app_name)
            if not nodes:
                continue

            if len(nodes) > 1:
                name_str = "; ".join(
                    "%s in %s" % (name, app) for app, name in nodes
                )
                raise CommandError(
                    "Conflicting migrations detected; multiple leaf nodes in the "
                    "migration graph: (%s).\nTo fix them run "
                    "'python manage.py makemigrations --merge'" % name_str
                )

            leaf_migrations[app_name] = graph.leaf_nodes(app_name)[0][1]

        result = json.dumps(
            leaf_migrations,
            sort_keys=True,
            indent=4,
            separators=(',', ': '),
        )

        if not os.path.exists(releases_dir):
            os.makedirs(releases_dir)

        with open(os.path.join(releases_dir, "{}.json".format(options['release'])), "w") as fp:
            fp.write(result)

rollback.py

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

import os
import json
import time

from django.conf import settings
from django.core.management.base import BaseCommand, CommandError
from django.db import DEFAULT_DB_ALIAS, connections
from django.db.migrations.executor import MigrationExecutor


class Command(BaseCommand):
    help = "Rollback a release."

    def add_arguments(self, parser):
        parser.add_argument(
            'release',
            help='Database state will be reverted to the state it was in at that release',
        )
        parser.add_argument(
            '--noinput',
            '--no-input',
            action='store_false',
            dest='interactive',
            default=True,
            help='Tells Django to NOT prompt the user for input of any kind.',
        )
        parser.add_argument(
            '--database',
            action='store',
            dest='database',
            default=DEFAULT_DB_ALIAS,
            help='Nominates a database to synchronize. Defaults to the "default" database.'
        )

    def handle(self, *args, **options):
        releases_dir = getattr(
            settings,
            "RELEASES_DIR",
            os.path.join(settings.PROJECT_PATH, "releases"),
        )

        release_path = os.path.join(releases_dir, "{}.json".format(options['release']))
        if not os.path.exists(release_path):
            raise CommandError('No release file {!r}'.format(release_path))

        with open(release_path) as fp:
            release = json.load(fp)
        targets = release.items()

        self.verbosity = options.get('verbosity')
        self.interactive = options.get('interactive')

        # Get the database we're operating from
        db = options.get('database')
        connection = connections[db]

        # Hook for backends needing any database preparation
        connection.prepare_database()
        # Work out which apps have migrations and which do not
        executor = MigrationExecutor(connection, self.migration_progress_callback)

        # Before anything else, see if there's conflicting apps and drop out
        # hard if there are any
        conflicts = executor.loader.detect_conflicts()
        if conflicts:
            name_str = "; ".join(
                "%s in %s" % (", ".join(names), app)
                for app, names in conflicts.items()
            )
            raise CommandError(
                "Conflicting migrations detected; multiple leaf nodes in the "
                "migration graph: (%s).\nTo fix them run "
                "'python manage.py makemigrations --merge'" % name_str
            )

        plan = executor.migration_plan(targets)

        if not len(plan):
            raise CommandError("Nothing to rollback")

        for migration, applied in plan:
            if not applied:
                raise CommandError(
                    "Migration {} would be applied rather than reverted. This does not make sense and may not be safe.".format(migration.name)
                )

        if self.verbosity >= 1:
            self.stdout.write(self.style.MIGRATE_HEADING("Operations to perform:"))
            for migration, applied in plan:
                self.stdout.write("  Revert {} {}".format(migration.app_label, migration.name))

        if self.verbosity >= 1:
            self.stdout.write(self.style.MIGRATE_HEADING("Running migrations:"))

        executor.migrate(targets, plan, fake=False, fake_initial=False)

    def migration_progress_callback(self, action, migration=None, fake=False):
        if self.verbosity >= 1:
            compute_time = self.verbosity > 1
            if action == "apply_start":
                if compute_time:
                    self.start = time.time()
                self.stdout.write("  Applying %s..." % migration, ending="")
                self.stdout.flush()
            elif action == "apply_success":
                elapsed = " (%.3fs)" % (time.time() - self.start) if compute_time else ""
                if fake:
                    self.stdout.write(self.style.MIGRATE_SUCCESS(" FAKED" + elapsed))
                else:
                    self.stdout.write(self.style.MIGRATE_SUCCESS(" OK" + elapsed))
            elif action == "unapply_start":
                if compute_time:
                    self.start = time.time()
                self.stdout.write("  Unapplying %s..." % migration, ending="")
                self.stdout.flush()
            elif action == "unapply_success":
                elapsed = " (%.3fs)" % (time.time() - self.start) if compute_time else ""
                if fake:
                    self.stdout.write(self.style.MIGRATE_SUCCESS(" FAKED" + elapsed))
                else:
                    self.stdout.write(self.style.MIGRATE_SUCCESS(" OK" + elapsed))
            elif action == "render_start":
                if compute_time:
                    self.start = time.time()
                self.stdout.write("  Rendering model states...", ending="")
                self.stdout.flush()
            elif action == "render_success":
                elapsed = " (%.3fs)" % (time.time() - self.start) if compute_time else ""
                self.stdout.write(self.style.MIGRATE_SUCCESS(" DONE" + elapsed))