An rclone backup script for cron

rclone-cron.sh

#!/bin/bash

##############################################################################
# An rclone backup script by Jared Males ([email protected])
# 
# Copyright (C) 2018 Jared Males <[email protected]>
#
# This script is licensed under the terms of the MIT license.
# https://opensource.org/licenses/MIT
#
# Runs the 'rclone sync' command.  Designed to be used as a cron job.
#
# 1) Backup Source
#    Edit the SRC variable below to point to the directory you want to backup.
#
# 2) Backup Destination
#    Edit the DEST variable to point to the remote and location (see rclone docs).
# 
# 3) Excluding files and directories
#    Edit the EXCLUDEFILE variable below to point to a file listing files and directories to exclude.
#    See the rclone docs for the format.
#
#    Also, any directory can be excluded by adding an '.rclone-ignore' file to it without editing the exclude file.
#    This file can be empty.  You can edit the name of this file with EXIFPRESENT below.
# 
# 4) You can change the bandwidth limits by editing BWLIMT, which includes a timetable facility.  
#    See rclone docs for more info.
#
# 5) Logs:
#    -- The output of rclone is written to the location specified by LOGFILE.  This is rotated with savelog. 
#       The details of synclog can be edited. 
#    -- The log rotation, and start and stop times of this script, are written to the location specified by CRONLOG.
#       This isn't yet rotated, probably should be based on size.
#   
##############################################################################

#### rclone sync options

SRC=/home/jrmales

#---- Edit this to the desired destination
DEST=b2:backup-mx9

#---- This is the path to a file with a list of exclude rules
EXCLUDEFILE=$SRC/.rclone/excludes 

#---- Name of exclude file
# NOTE: you need "v1.39-036-g2030dc13β" or later for this to work.
EXIFPRESENT=.rclone-ignore

#---- The bandwidth time table
BWLIMIT="08:00,512 00:00,off"

#---- Don't sync brand new stuff, possible partials, etc.
MINAGE=15m

#---- [B2 Specific] number of transfers to do in parallel.  rclone docs say 32 is recommended for B2.
TRANSFERS=32

#---- Location of sync log [will be rotated with savelog]
LOGFILE=$SRC/.rclone/rclone-sync.log
LOGS='-vv --log-file='$LOGFILE

#---- Location of cron log
CRONLOG=$SRC/.rclone/rclone-cron.log


###################################################
## Locking Boilerplate from https://gist.github.com/przemoc/571091
## Included under MIT License:
###################################################

## Copyright (C) 2009 Przemyslaw Pawelczyk <[email protected]>
##
## This script is licensed under the terms of the MIT license.
## https://opensource.org/licenses/MIT
#
# Lockable script boilerplate

### HEADER ###

LOCKFILE="/tmp/`basename $0`"
LOCKFD=99

# PRIVATE
_lock()             { flock -$1 $LOCKFD; }
_no_more_locking()  { _lock u; _lock xn && rm -f $LOCKFILE; }
_prepare_locking()  { eval "exec $LOCKFD>\"$LOCKFILE\""; trap _no_more_locking EXIT; }

# ON START
_prepare_locking

# PUBLIC
exlock_now()        { _lock xn; }  # obtain an exclusive lock immediately or fail
exlock()            { _lock x; }   # obtain an exclusive lock
shlock()            { _lock s; }   # obtain a shared lock
unlock()            { _lock u; }   # drop a lock

###################################################
# End of locking code from Pawelczyk
###################################################


#make a log entry if we exit because locked
exit_on_lock()      { echo $(date -u)' | rclone-cron.sh already running.' >> $CRONLOG; exit 1; }


#Now check for lock
exlock_now || exit_on_lock
#We now have the lock.

#Rotate logs.
savelog -n -c 7 $LOGFILE >> $CRONLOG

#Log startup
echo $(date -u)' | starting rclone-cron.sh . . .' >> $CRONLOG

#Now do the sync!
rclone sync $SRC $DEST --transfers $TRANSFERS --bwlimit "$BWLIMIT" --min-age $MINAGE --exclude-from $EXCLUDEFILE --exclude-if-present $EXIFPRESENT --delete-excluded $LOGS


#log success
echo $(date -u)' | completed rclone-cron.sh.' >> $CRONLOG

#release the lock
unlock


exit

have some dependencies? paste your bash but savelog is not present:

line 114: savelog: command not found

Some things to note here:

1. bandwidth limit for rclone is in units of bytes/sec not bits/sec. This isn't the problem of the script but it's worth noting if you're setting up bandwidth limits throughout the day (10 MB/s is very different to 10 Mbps!)

2. I couldn't get the exclude file to work. Commenting out and removing the exclude lines works just fine, since we're looking for a total backup

3. It's possible to backup to two different backups simultaneously using the & command as you would in any linux command line. However, the bandwidth limit applies to EACH sync command, so if you set a bandwidth limit of 0.625M and have two simultaneous backups/syncs, then total system bandwidth will be 1.25M

Hello,

I tried to use this in debian (RPI) but get always in line 131 an syntax error "rclone-cron.sh: line 131: syntax error: unexpected end of file (expecting "}")
I am sorry, I am not very good in shell script, and I do not find the problem....

Thank you

Ragards
Thorsten

Hi,

Sorry it didn't work.

Hello,

I tried to use this in debian (RPI) but get always in line 131 an syntax error "rclone-cron.sh: line 131: syntax error: unexpected end of file (expecting "}")

I just tested this, copy and pasted from here, on a new Ubuntu 20 and a Raspbian install and it worked. I'm not sure what would cause your error, except maybe a copy and paste problem, like a line getting truncated (?)

All of the {} pairs are one line so make sure they are matched.

I am sorry, I am not very good in shell script, and I do not find the problem....

Thank you

Ragards
Thorsten

Thank you for trying to help.

I found the problem.....I had windows line ends...after changing my editor to uniy line ends it runs.

Regards

thank you so much for this script. it was very useful.

for others who may have used it or are about to use it, i encountered a few problems in setting it up. this is what i did to resolve the problems.

i originally set it up on an old test server running centos 6.9, so some problems may have occurred as a result of that.

i encountered a line ending problem as a result of editing it on a windows machine
/bin/bash^M: bad interpreter: No such file or directory
i had to edit the file in vim instead

there was also an error in the log:
Config file "/.config/rclone/rclone.conf" not found - using defaults
this is solved by specifying the config file, appending this to the rclone call:
--config [/path/to/file]

finally, i got output which indicated the savelog could not be found.
i wasn't able to source savelog for my server, probably because it is no longer supported.
i dare say when i use it on other up to date servers i will not have a problem

thanks again

ps: i also had to add --copy-links to the command to ensure that symlinks are copied into my backup
i also ended up using logrotate instead of savelog to turn over the logs
finally, be aware that this script also enables verbose logging. unless you want excessive logging detail, it may be worth experimenting with different logging levels. i ended up removing the level indicator which meant it is using the default level. all problems are logged with this
LOGS='--log-file='$LOGFILE

hope all this helps someone

Thanks @raystrach

Thanks for sharing this useful script Jared.

Perhaps you can add an 'OPTIONS' variable for adding additional options (command line switches to rclone) and/or add --fast-list to it to speed up the sync and lower cost (due to much less Class C transactions on backblaze)

https://rclone.org/docs/#fast-list
https://forum.rclone.org/t/syncing-to-backlaze-b2-extremely-slow-unless-i-enable-fast-list/17222

Can you add multiple sources to same destination? Like "src1","src2" or "src1""src2" or something similar?

@CPC-STaylor a quick read of rclone docs says no, not without adding a lot to this code. rclone sync appears to only take a single path at a time. It might be simplest to make SRC an argument and call this from a separate script.

Just created an account to say thank you for this script. I was looking around for something that was a bit more than a single line, but didn't need the complexity of rclone jobber. Modified it to utilize copy instead of sync and it does exactly what I needed.