A method of backing up your website to Amazon S3.

backup-to-s3.sh

#!/bin/bash

## Email Variables
EMAILDATE=`date --date="today" +%y-%m-%d`
EMAIL="[email protected]"

SUBJECT="[servername] Backup Script Started! - "$EMAILDATE
EMAILMESSAGE="/tmp/emailmessage1.txt"
echo "Just to let you know that the backup script has started."> $EMAILMESSAGE
/bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGE

# Set up the variables
### The URI of the S3 bucket.
S3URI='s3://bucketname/'

### An array of directories you want to backup (I included a few configuration directories to).
DirsToBackup=(
'/var/www/vhosts/domain1.com/httpdocs'
'/var/www/vhosts/domain2.com/httpdocs'
'/var/www/vhosts/domain3.com/httpdocs'
)

### The databases you want to backup
DBsToBackup=(
'database1'
'database2'
'database3'
)

### The directory we're going to story our backups in on this server.
TmpBackupDir='/path/to/temp/s3backups/folder'


## The MySQL details
MySQLDetails[0]='localhost' # MySQL Host
MySQLDetails[1]='backup_user' # User
MySQLDetails[2]='yourstrongpassword' # Password


## The expiry dates of the backups
### Only store 0 days of backups on the server.
### Changed to 0 days to not fill the server with unneccessary backups
Expiry[0]=`date --date="today" +%y-%m-%d`

### Only store 2 weeks worth of backups on S3 
Expiry[1]=`date --date="2 weeks ago" +%y-%m-%d`

### Using ExpiryDayOfMonth to skip first day of the month when deleting so monthly backups are kept on s3
ExpiryDayOfMonth=`date --date="2 weeks ago" +%d`

### Todays date.
TodayDate=`date --date="today" +%y-%m-%d`

## Finally, setup the today specific variables.
Today_TmpBackupDir=$TmpBackupDir'/'$TodayDate


# Start backing up things.

## Check we can write to the backups directory
if [ -w "$TmpBackupDir" ]
then
  # Do nothing and move along.
    echo 'Found and is writable:  '$TmpBackupDir
else
    echo "Can't write to: "$TmpBackupDir
    exit
fi

## Make the backup directory (Also make it writable)
echo ''
echo 'Making Directory: '$Today_TmpBackupDir
mkdir $Today_TmpBackupDir
chmod 0777 $Today_TmpBackupDir

## GZip the directories and put them into the backups folder
echo ''
for i in "${DirsToBackup[@]}"
do
    filename='dir-'`echo $i | tr '/' '_'`'.tar.gz'
    echo 'Backing up '$i' to '$Today_TmpBackupDir'/'$filename
    tar -czpPf $Today_TmpBackupDir'/'$filename $i
done

## Backup the MySQL databases
echo ''
for i in "${DBsToBackup[@]}"
do
    filename='mysql-'$i'.sql'
    echo 'Dumping DB '$i' to '$Today_TmpBackupDir'/'$filename
    mysqldump -h "${MySQLDetails[0]}" -u "${MySQLDetails[1]}" -p"${MySQLDetails[2]}" $i > $Today_TmpBackupDir'/'$filename
    tar -czpPf $Today_TmpBackupDir'/'$filename'.tar.gz' $Today_TmpBackupDir'/'$filename
    rm -R $Today_TmpBackupDir'/'$filename
done

## Alert admin that backup complete, starting sync
SUBJECT="[servername] Backup Complete, Starting Sync! - "$EMAILDATE
EMAILMESSAGE="/tmp/emailmessage2.txt"
echo "Just to let you know that the backup script has finished and we're starting sync to s3 now."> $EMAILMESSAGE
/bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGE

## Sending new files to S3
echo ''
echo 'Syncing '$Today_TmpBackupDir' to '$S3URI$TodayDate'/'
s3cmd put --recursive $Today_TmpBackupDir $S3URI
if [ $? -ne 0 ]; then
    SUBJECT="s3cmd put failed on [servername]"
    EMAILMESSAGE="/tmp/emailmessage3.txt"
        echo "Just to let you know that the s3cmd put of '$Today_TmpBackupDir' failed."> $EMAILMESSAGE
        echo "You should check things out immediately." >>$EMAILMESSAGE
    /bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGE
fi

# Cleanup.
echo ''
echo 'Removing local expired backup: '$TmpBackupDir'/'${Expiry[0]}
rm -R $TmpBackupDir'/'${Expiry[0]}

if [ "$ExpiryDayOfMonth" != '01' ]; then
    echo 'Removing remote expired backup: '$S3URI${Expiry[1]}'/'
    s3cmd del $S3URI${Expiry[1]}'/' --recursive
else
    echo 'No need to remove backup on the 1st'
fi

echo 'Making '$Today_TmpBackupDir' permissions 0755'
chmod 0755 $Today_TmpBackupDir

echo 'All Done! Yay! (",)'

## Notify admin that the script has finished
SUBJECT="[servername] S3 Sync Complete! - "$EMAILDATE
EMAILMESSAGE="/tmp/emailmessage4.txt"
echo "Just to let you know that the s3 sync has now completed."> $EMAILMESSAGE
/bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGE


## Email Report of What Exists on S3 in Today's Folder
exec 1>'/tmp/s3report.txt'
s3cmd ls s3://bucketname/$TodayDate/

SUBJECT="S3 Backup Report of [servername]: "$TodayDate
EMAILMESSAGE="/tmp/s3report.txt"
/bin/mail -s "$SUBJECT" "$EMAIL" < $EMAILMESSAGE

createBackupUser.sql

CREATE USER 'backup_user'@'localhost' IDENTIFIED BY 'SomeSecurePassword!';
GRANT SELECT, LOCK TABLES ON *.* TO 'backup_user'@'localhost';

cronExample

0 3    * * *   bash    /path/to/your/backup-to-s3.sh

Hi,
I am trying to understand how this works:
### Using ExpiryDayOfMonth to skip first day of the month when deleting so monthly backups are kept on s3
ExpiryDayOfMonth=date --date="2 weeks ago" +%d

I am trying to do a weekly backup using your script. Will this still work to save one copy every month on S3?
Thanks!

THANKS for this script! Just today, 2015-01-01, used this on a relatively new AMI EC2 to backup my mysql databases and websites. Instead of s3cmd, which is not available by default on AMI Linux, I just replaced with those lines with the newer "aws s3" command. For example, instead of:

s3cmd put --recursive $Today_TmpBackupDir $S3URI

I used:

aws s3 cp $Today_TmpBackupDir $S3URI$TodayDate --recursive --profile my-backup-user

Similarily, I modified the "del" and "ls" command lines in the script using:
aws s3 rm
aws s3 ls

Documentation here for the aws s3 CLI:
http://docs.aws.amazon.com/cli/latest/reference/s3/index.html
Note that you must configure an aws access key and secret for use with the AWS CLI tools. The docs were confusing to me, but this helped me through it:
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html

I believe the script has a couple of small bugs. The first and most important is this line:
s3cmd put --recursive $Today_TmpBackupDir $S3URI

I believe is supposed to be this:
s3cmd put --recursive $Today_TmpBackupDir $S3URI$TodayDate

Without that last bit, you won't have daily directories in your S3 bucket.

The second more minor change is this line:
s3cmd ls s3://bucketname/$TodayDate/

I assume was supposed to be this:
s3cmd ls $S3URI$TodayDate/

So I ended up changing the 3 S3 commands to these:
aws s3 cp $Today_TmpBackupDir $S3URI$TodayDate --recursive --profile my-backup-user
aws s3 rm $S3URI${Expiry[1]}'/' --recursive --profile my-backup-user
aws s3 ls $S3URI$TodayDate/ --profile my-backup-user

Note that mail does not work by default on EC2 AMI instances, so for now, I have the mail lines commented out in my script, but it looks like I might use the AWS SNS service for that notification functionality.

Hello;

Can i exclude the MYSQL backup options? also i want a weekly backup to S3

Retention is not working, in details script in not able to remove the older backups

I have change few things in the script as follows

change the date format from +%y-%m-%d to +%d-%m-%y

can you please help me out in it or do i have to change the date as mention in the script

This is not working
s3cmd del $S3URI${Expiry[1]}'/' --recursive

Moreover echoing is also not display expiray
echo 'Removing remote expired backup: '$S3URI${Expiry[1]}'/'

I need help

Regards
Syed Ejaz