MAAS: Backup & Restore

maas-backup-and-restore.md

Dumping

Use the directory output format because it works better with incremental backups and is most flexible when restoring.

Do not compress because the overwhelming majority of the contents of the MAAS database is already compressed images. For a MAAS installation with one image the saving due to compression was only 1.7% but the dump took noticeably longer. With more images the ratio of already-compressed to not-compressed data goes up, and the savings will likely drop.

From the command-line something like the following works well:

sudo -u postgres \
    pg_dump --format=d --file=$dumpdir \
        --verbose --compress=0 maasdb

Note that $dumpdir should not exist before running this command.

Programmatically this can be done something like:

export PGHOST=localhost
export PGUSER=maas

touch temporary_pgpass
chmod 600 temporary_pgpass
# host:port:database:username:password
echo '*:*:*:*:t8A8Tc7Py3bG' > temporary_pgpass
export PGPASSFILE=temporary_pgpass

pg_dump --format=d --file=$dumpdir --no-password \
    --compress=0 --verbose maasdb

rm -f temporary_pgpass

with values obtained from /etc/maas/regiond.conf. Do not actually use this code; it's only to show the steps needed.

Restoring

A new database may be required into which the dump can be loaded. This cannot be done via the maas role because it does not have permission to create databases. We could consider granting this.

sudo -u postgres createdb --no-password maasdb2

Then the restore can be done using the maas role, with no need for elevated privileges:

export PGHOST=localhost
export PGUSER=maas

touch temporary_pgpass
chmod 600 temporary_pgpass
# host:port:database:username:password
echo '*:*:*:*:t8A8Tc7Py3bG' > temporary_pgpass
export PGPASSFILE=temporary_pgpass

pg_restore --dbname=maasdb2 --single-transaction $dumpdir

rm -f temporary_pgpass

(Note that --single-transaction implies --exit-on-error.)

However this will not work. A less robust pg_restore command is needed:

pg_restore --dbname=maasdb2 $dumpdir

The reason is that the following error occurs when restoring:

pg_restore: [archiver (db)] Error while PROCESSING TOC:
pg_restore: [archiver (db)] Error from TOC entry 3770; 0 0 COMMENT EXTENSION plpgsql
pg_restore: [archiver (db)] could not execute query: ERROR:  must be owner of extension plpgsql
    Command was: COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';

This is a known issue upstream.

This problem can be worked around by editing the restore list to comment out the broken step, then we can use --single-transaction again:

pg_restore --list $dumpdir > restore.list
sed -i '/ COMMENT - EXTENSION /s/^/;/' restore.list
pg_restore --dbname=maasdb2 --single-transaction \
    --use-list=restore.list $dumpdir

This completes okay with a few warnings:

pg_restore: WARNING:  no privileges could be revoked for "public"
pg_restore: WARNING:  no privileges could be revoked for "public"
pg_restore: WARNING:  no privileges were granted for "public"
pg_restore: WARNING:  no privileges were granted for "public"

??? What happens in a pristine new cluster that does not yet have the PL/pgSQL extension? Is that a thing?

Post-restore

If this database is to be used to run MAAS, the shared secret on the filesystem needs to match that in the database. This gets complicated because the shared secret is also stored in the Debian control database (or whatever it's called). We need a mechanism to say: get the shared secret from the database, overwrite whatever is on the filesystem, and also update the packaging database. Note that where no secret exists on the filesystem we can simply start MAAS and it will DTRT.

QA

1. Install MAAS.

2. Run smoke tests.

3. Dump database.

4. Restore from dump.

5. Switch MAAS to restored database: sudo maas-region local_config_set --database-name=$restored_name

6. Restart maas-regiond.service.

7. Run smoke tests.

We may also want to restore to a pristine machine where MAAS has only just been installed.