Skip to content

Instantly share code, notes, and snippets.

@paolodina

paolodina/.obnam.conf

Last active August 29, 2015 14:08
Show Gist options
  • Save paolodina/563dd42262af08015fe7 to your computer and use it in GitHub Desktop.
Save paolodina/563dd42262af08015fe7 to your computer and use it in GitHub Desktop.
Download ZIP
Obnam backup script + LVM Snapshot
Raw
.obnam.conf
# current settings
# obnam --dump-config
# Docs:http://code.liw.fi/obnam/manual/obnam-manual.en.html
# ###########################
# Use your values
[config]
repository = /media/paolo/905d8628-93c9-44d3-9619-02693334cc03/obnam-backup
exclude-caches = yes
one-file-system = yes
log = /home/paolo/backup.obnam.log
# http://unix.stackexchange.com/a/150851/8115
lru-size = 1024
upload-queue-size = 512
Raw
requirements.txt
# doesn't work
#http://code.liw.fi/debian/pool/main/g/genbackupdata/genbackupdata_1.7.orig.tar.gz
#error: can't copy 'genbackupdata.1': doesn't exist or not a regular file
#http://code.liw.fi/debian/pool/main/s/summain/summain_0.19.orig.tar.gz
# error: can't copy 'summain.1': doesn't exist or not a regular file
paramiko
http://code.liw.fi/debian/pool/main/p/python-cliapp/python-cliapp_1.20140719.orig.tar.gz
http://code.liw.fi/debian/pool/main/p/python-tracing/python-tracing_0.8.orig.tar.gz
http://code.liw.fi/debian/pool/main/p/python-ttystatus/python-ttystatus_0.23.orig.tar.gz
http://code.liw.fi/debian/pool/main/c/cmdtest/cmdtest_0.12.orig.tar.gz
http://code.liw.fi/debian/pool/main/s/seivot/seivot_1.18.orig.tar.gz
# sudo apt-get install libfuse-dev
https://pypi.python.org/packages/source/f/fuse-python/fuse-python-0.2.tar.gz
Raw
requirements2.txt
http://code.liw.fi/debian/pool/main/p/python-larch/python-larch_1.20131130.orig.tar.gz
http://code.liw.fi/debian/pool/main/o/obnam/obnam_1.8.orig.tar.gz
Raw
runbackup.sh
#!/bin/sh
: <<'end_readme'
======================
Obnam backup for linux
======================
DISCLAIMER
==========
This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY expressed or implied, including the implied warranties of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Requirements
============
* obnam - http://obnam.org/
* python
* virtualenv - python package to manage standalone local installations
* pip - python package installer
Install
=======
Create and activate a new virtual environment::
$ virtualenv ~/path/venvs/obnam && ~/path/venvs/obnam/bin/activate
install obnam and other python deps::
$ pip install -U -r ./requirements/requirements.txt # see attached file
$ pip install -U -r ./requirements/requirements2.txt # see attached file
Setup
=====
Create a top level /obnam-backup directory. It's used as a root directory to
mount LVM snapshots (if used) before running the backup::
sudo mkdir /obnam-backup
You can customize this name, just update the relative setting below.
NOTES
=====
The script is configured to use LVM snapshots to take a consistent copy
of the data. A better approach would be to rely on filesystems where
snapshots are natively supported, like ZFS or brtfs. It's even possible (but
not suggested) to take backups without using snapshots at all.
Ref.
====
* http://code.liw.fi/obnam/manual/obnam-manual.en.html
* https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/index.html
* http://www.tldp.org/HOWTO/LVM-HOWTO/
* http://hydra.geht.net/tino/howto/linux/lvm2/harmful/
TODO
====
1. convert bash scripts to python (https://pypi.python.org/pypi/sh)
2. add support for config file management
3. ncurses interface (http://npyscreen.readthedocs.org/index.html)
4. cli interface (http://click.pocoo.org/)
5. pre/post hooks
6. https://pypi.python.org/pypi/python-crontab/
7. bare metal restore mini distro
end_readme
# ###############################################################
# Customize these settings.
# ###############################################################
# The obnam virtual environment directory
VENV_HOME=/home/paolo/local2/venvs/obnam
# TODO: ^- factor out to separate config file
# ###############################################################
# Root mountpoint where snapshot are mounted
SNAPSHOT_MOUNT=/backup-snapshots && mkdir -p $SNAPSHOT_MOUNT
# TODO: ^- factor out this setting to separate config file
# ###############################################################
# MY PARTITIONS, DISK USAGE AND BACKUP SCHEME ARE PROBABLY
# DIFFERENT THAT YOURS. CHANGE THE FOLLOWING DEFINITIONS
# ACCORDINGLY.
# ###############################################################
# MY / partition at a certain time ###########################
# Filesystem Size Used Avail Use% Mounted on
# /dev/mapper/sysvg-lvroot 9,8G 2,3G 7,0G 25% /
#
# LV Path /dev/sysvg/lvroot
# LV Name lvroot
# VG Name sysvg
#
# I want the snapshot size to be 75% of total origin space (YMMV).
# 9.8G * 0.75% = 7.35G
#
# No really, thanks to copy on write (cow), snapshots require much less space
# than the space dedicated to live data. Considering that backup snapshots have
# a short live and that during backup live data won't change much, set a
# smaller value.
#
# Note 1: in the example above, /dev/mapper/sysvg-lvroot and /dev/sysvg/lvroot
# are symlink pointing to the same target (at least in Ubuntu).
#
# Note 2: other filesystems are mounted over the / filesystem, therefore it
# must be the first in the list.
SNAPSHOT_ROOT_MOUNT=$SNAPSHOT_MOUNT
SNAPSHOT_ROOT_NAME=backup_snapshot_root
LV_ROOT_MAPPER=/dev/mapper/sysvg-lvroot
echo Create into sysvg a snapshot of lvroot named backup_snapshot_root
lvcreate --size 3G --snapshot --name $SNAPSHOT_ROOT_NAME $LV_ROOT_MAPPER
sleep 1
echo Mounting root snapshot to filesystem...
mount -o ro /dev/mapper/sysvg-$SNAPSHOT_ROOT_NAME $SNAPSHOT_ROOT_MOUNT
sleep 1
# ###############################################################
# MY /home partition at a certain time ##########################
# Filesystem Size Used Avail Use% Mounted on
# /dev/mapper/sysvg-lvhome 246G 93G 142G 40% /home
#
# LV Path /dev/sysvg/lvhome
# LV Name lvhome
# VG Name sysvg
#
# I want the snapshot size to be 75% of total origin used space (YMMV).
# 93G * 0.75% = 69.75G
#
# No really, thanks to copy on write (cow), snapshots require much less space
# than the space dedicated to live data. Considering that backup snapshots have
# a short live and that during backup live data won't change much, set a
# smaller value.
#
# Note 1: in the example above, /dev/mapper/sysvg-lvhome and /dev/sysvg/lvhome
# are symlink pointing to the same target (at least in Ubuntu).
SNAPSHOT_HOME_MOUNT=$SNAPSHOT_MOUNT/home
SNAPSHOT_HOME_NAME=backup_snapshot_home
LV_HOME_MAPPER=/dev/mapper/sysvg-lvhome
echo Create into sysvg a snapshot of lvhome named backup_snapshot_home
lvcreate --size 50G --snapshot --name $SNAPSHOT_HOME_NAME $LV_HOME_MAPPER
sleep 1
echo Mounting home snapshot to filesystem...
mount -o ro /dev/mapper/sysvg-$SNAPSHOT_HOME_NAME $SNAPSHOT_HOME_MOUNT
sleep 1
# ###############################################################
# MY /var partition at a certain time ###########################
# Filesystem Size Used Avail Use% Mounted on
# /dev/mapper/sysvg-lvvar 2,9G 1,1G 1,7G 40% /var
#
# LV Path /dev/sysvg/lvvar
# LV Name lvvar
# VG Name sysvg
#
# I want the snapshot size to be 75% of total origin space (YMMV).
# 2.9G * 0.75% = 2.175G
#
# No really, thanks to copy on write (cow), snapshots require much less space
# than the space dedicated to live data. Considering that backup snapshots have
# a short live and that during backup live data won't change much, set a
# smaller value.
#
# Note 1: in the example above, /dev/mapper/sysvg-lvvar and /dev/sysvg/lvvar
# are symlink pointing to the same target (at least in Ubuntu).
SNAPSHOT_VAR_MOUNT=$SNAPSHOT_MOUNT/var
SNAPSHOT_VAR_NAME=backup_snapshot_var
LV_VAR_MAPPER=/dev/mapper/sysvg-lvvar
echo Create into sysvg a snapshot of lvvar named backup_snapshot_var
lvcreate --size 1G --snapshot --name $SNAPSHOT_VAR_NAME $LV_VAR_MAPPER
sleep 1
echo Mounting home snapshot to filesystem...
mount -o ro /dev/mapper/sysvg-$SNAPSHOT_VAR_NAME $SNAPSHOT_VAR_MOUNT
sleep 1
# ###############################################################
# MY boot partition at a certain time ###########################
# Filesystem Size Used Avail Use% Mounted on
# /dev/mapper/sysvg-lvboot 2,0G 243M 1,6G 14% /boot
#
# LV Path /dev/sysvg/lvboot
# LV Name lvboot
# VG Name sysvg
#
# I want the snapshot size to be 75% of total origin space (YMMV).
# 2G * 0.75% = 1.5G
#
# No really, thanks to copy on write (cow), snapshots require much less space
# than the space dedicated to live data. Considering that backup snapshots have
# a short live and that during backup live data won't change much, set a
# smaller value.
#
# Note: in the example above, /dev/mapper/sysvg-lvboot and /dev/sysvg/lvboot are
# symlink pointing to the same target (at least in Ubuntu).
SNAPSHOT_BOOT_MOUNT=$SNAPSHOT_MOUNT/boot
SNAPSHOT_BOOT_NAME=backup_snapshot_boot
LV_BOOT_MAPPER=/dev/mapper/sysvg-lvboot
echo Create into sysvg a snapshot of lvboot named backup_snapshot_boot
lvcreate --size 500M --snapshot --name $SNAPSHOT_BOOT_NAME $LV_BOOT_MAPPER
sleep 1
echo Mounting boot snapshot to filesystem...
mount -o ro /dev/mapper/sysvg-$SNAPSHOT_BOOT_NAME $SNAPSHOT_BOOT_MOUNT
sleep 1
# ###############################################################
# MY /usr partition at a certain time ###########################
# Filesystem Size Used Avail Use% Mounted on
# /dev/mapper/sysvg-lvusr 9,8G 4,4G 4,9G 48% /usr
#
# LV Path /dev/sysvg/lvusr
# LV Name lvusr
# VG Name sysvg
#
# I want the snapshot size to be 75% of total origin space (YMMV).
# 9.8G * 0.75% = 7.35G
#
# No really, thanks to copy on write (cow), snapshots require much less space
# than the space dedicated to live data. Considering that backup snapshots have
# a short live and that during backup live data won't change much, set a
# smaller value.
#
# Note 1: in the example above, /dev/mapper/sysvg-lvvar and /dev/sysvg/lvvar
# are symlink pointing to the same target (at least in Ubuntu).
SNAPSHOT_USR_MOUNT=$SNAPSHOT_MOUNT/usr
SNAPSHOT_USR_NAME=backup_snapshot_usr
LV_USR_MAPPER=/dev/mapper/sysvg-lvusr
echo Create into sysvg a snapshot of lvusr named backup_snapshot_usr
lvcreate --size 3G --snapshot --name $SNAPSHOT_USR_NAME $LV_USR_MAPPER
sleep 1
echo Mounting usr snapshot to filesystem...
mount -o ro /dev/mapper/sysvg-$SNAPSHOT_USR_NAME $SNAPSHOT_USR_MOUNT
sleep 1
# ###############################################################
# Backup ########################################################
# Order of partitions to backup is customizable.
echo run backup...
$VENV_HOME/bin/python \
$VENV_HOME/bin/obnam backup
$SNAPSHOT_BOOT_MOUNT \
$SNAPSHOT_ROOT_MOUNT \
$SNAPSHOT_USR_MOUNT \
$SNAPSHOT_VAR_MOUNT
$SNAPSHOT_HOME_MOUNT
# ###############################################################
# Cleanup #######################################################
echo home snapshot umount
umount $SNAPSHOT_HOME_MOUNT
sleep 1
echo usr snapshot umount
umount $SNAPSHOT_USR_MOUNT
sleep 1
echo var snapshot umount
umount $SNAPSHOT_VAR_MOUNT
sleep 1
echo boot snapshot umount
umount $SNAPSHOT_BOOT_MOUNT
sleep 1
# and always for last...
echo snapshot umount
umount $SNAPSHOT_ROOT_MOUNT
sleep 1
echo remove home snapshot
lvremove -f /dev/mapper/sysvg-$SNAPSHOT_HOME_NAME
sleep 1
echo remove usr snapshot
lvremove -f /dev/mapper/sysvg-$SNAPSHOT_USR_NAME
sleep 1
echo remove var snapshot
lvremove -f /dev/mapper/sysvg-$SNAPSHOT_VAR_NAME
sleep 1
echo remove boot snapshot
lvremove -f /dev/mapper/sysvg-$SNAPSHOT_BOOT_NAME
sleep 1
echo remove root snapshot
lvremove -f /dev/mapper/sysvg-$SNAPSHOT_ROOT_NAME
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment