Autorun Synology Hyper Backup and Integrity Check with Email Notifications

autorun.sh

#!/bin/sh

# This script is to be used in combination with Synology Autorun:
# - https://github.com/reidemei/synology-autorun
# - https://github.com/Jip-Hop/synology-autorun
#
# You need to change the task_id to match your Hyper Backup task.
# Get it with command: more /usr/syno/etc/synobackup.conf
#
# I like to keep "Beep at start and end" disabled in Autorun, because I don't
# want the NAS to beep after completing (could be in the middle of the night)
# But beep at start is a nice way to confirm the script has started,
# so that's why this script starts with a beep.
#
# After the backup completes, the integrity check will start. 
# Unfortunately in DSM you can't choose to receive email notifications of the integrity check results.
# So there's a little workaround, at the end of this script, to send an (email) notification.
# The results of the integrity check are taken from the synobackup.log file.
#
# In DSM -> Control Panel -> Notification I enabled email notifications,
# I changed its Subject to %TITLE% and the content to:
# Dear user,
#
# Integrity check for %TASK_NAME% is done.
#
# %OUTPUT%
#
# This way I receive email notifications with the results of the Integrity Check.
#
# Credits:
# - https://github.com/Jip-Hop
# - https://bernd.distler.ws/archives/1835-Synology-automatische-Datensicherung-mit-DSM6.html
# - https://www.beatificabytes.be/send-custom-notifications-from-scripts-running-on-a-synology-new/

task_id=6 # Hyper Backup task id, get it with command: more /usr/syno/etc/synobackup.conf
task_name="USB3 3TB Seagate" # Only used for the notification

/bin/echo 2 > /dev/ttyS1 # Beep on start

startTime=$(date +"%Y/%m/%d %H:%M:%S") # Current date and time

device=$2 # e.g. sde1, passed to this script as second argument

# Backup
/usr/syno/bin/synobackup --backup $task_id --type image

while sleep 60 && /var/packages/HyperBackup/target/bin/dsmbackup --running-on-dev $device
do
    :
done

# Check integrity
/var/packages/HyperBackup/target/bin/detect_monitor -k $task_id -t -f -g
# Wait a bit before detect_monitor is up and running
sleep 60
# Wait until check is finished, poll every 60 seconds
/var/packages/HyperBackup/target/bin/detect_monitor -k $task_id -p 60

# Send results of integrity check via email (from last lines of log file)

IFS=''
output=""
title=
NL=$'\n'

while read line
do
    
    # Compute the seconds since epoch for the start date and time
    t1=$(date --date="$startTime" +%s)
    
    # Date and time in log line (second column)
    dt2=$(echo "$line" | cut -d$'\t' -f2)
    # Compute the seconds since epoch for log line date and time
    t2=$(date --date="$dt2" +%s)
    
    # Compute the difference in dates in seconds
    let "tDiff=$t2-$t1"
    
    # echo "Approx diff b/w $startTime & $dt2 = $tDiff"
    
    # Stop reading log lines from before the startTime
    if [[ "$tDiff" -lt 0 ]]; then
        break
    fi
    
    text=`echo "$line" | cut -d$'\t' -f4`
    # Get rid of [Local] prefix
    text=`echo "$text" | sed 's/\[Local\]//'`
    
    if [ -z ${title} ]; then
        title=$text
    fi
    
    output="$output${NL}$text"
    
done <<<$(tac /var/log/synolog/synobackup.log)

# Hijack the ShareSyncError event to send custom message.
# This event is free to reuse because I don't use the Shared Folder Sync (rsync) feature.
# More info on sending custom (email) notifications: https://www.beatificabytes.be/send-custom-notifications-from-scripts-running-on-a-synology-new/
/usr/syno/bin/synonotify "ShareSyncError" "{\"%OUTPUT%\": \"${output}\", \"%TITLE%\": \"${title}\", \"%TASK_NAME%\": \"${task_name}\"}"

# Sleep a bit more before unmounting the disk
sleep 60

# Unmount the disk
exit 100

I found the above scripts very helpful in writing a version for my own purposes, which I ended up completely over-engineering. I originally wanted to support running multiple integrity checks in sequence against multiple HyperBackup jobs with targets on more than one external drive. In the end I also arbitrarily decided I only wanted to have to pass a single argument matching a HyperBackup job name and have multiple instances of the script figure out the rest. Pretty sure I made it way too complicated, but just in case anyone finds it useful I'll post it here anyway.

If you use this, be sure to schedule schedule a job to mount your drive(s), then one job for each HyperBackup task you want to check that calls this script with a matching task name argument, and then your actual backup jobs with the setting to remove the destination external device when the backup has completed selected, in that order. Synology queues HyperBackup jobs if they are scheduled while another is already running, so if you have multiple backup tasks targeting one disk, like me, just schedule them in the proper order one right after the other and only set the final one accessing the target disk to remove it.

There is definitely a fair bit of cleanup I could do and improvements I could make, plus there's a less than perfect locking mechanism and some ideas I had for multiple disk support in the functions ended up being unnecessary but left in anyway for now. Oh well.

Click for code

#!/bin/bash

# This script is to be called by a Scheduler task as root user,
# having 'Run command / User-defined script' filled in with your script's path.
# i. e. /bin/bash /volume1/Scripts/autointegrity.sh "<task string>"
#
# You need to pass a task string to match against.
# This may be a partial string, but must match a HyberBackup job name uniquely.
#
# I like to keep "Beep at start and end" disabled in Autorun, because I don't
# want the NAS to beep after completing (could be in the middle of the night)
# But beep at start is a nice way to confirm the script has started,
# so that's why this script starts with a beep.
#
# The script will wait until any drives are unmounted (via the Synology backup task)
# then re-mount the disk(s) and begin the integrity check(s).
# If you want to receive the log entries in an e-mail after this script finished,
# check 'Send run details by email' and fill in 'Email' in the Scheduler task settings.
#
# Tested with DSM 7.2.1-69057 Update 4 and Hyper Backup 4.1.0-3718 on a DS1821+.
#
# Credits:
# - https://gist.github.com/gersteba/6b07be49aa94c8df1bb88e7db615987d
# - https://gist.github.com/Jip-Hop/b9ddb2cc124302a5558659e1298c36ec
# - https://derwebprogrammierer.at/

function integrity_process_is_active() {
	# ------------------------------------------------------------------------------------------------
	# Function: integrity_process_is_active()
	# Purpose : Determine an integrity check is already running, with or without specific task_id
	# Returns : 0=Running, 1=Not running
	# ------------------------------------------------------------------------------------------------

	local _task_id

	if (( $# == 0 )); then
		_task_id=""
	else
		_task_id=" $1"
	fi

	# Find process(es)
	/bin/ps aux | /bin/grep -v grep | /bin/grep --quiet "detect_monitor \-\-task\-id${_task_id}"
}

function integrity_script_waiting() {
	# ------------------------------------------------------------------------------------------------
	# Function: integrity_script_waiting()
	# Purpose : Determine if other instances of this script are waiting to run integrity checks
	# Returns : 0=Waiting, 1=Not waiting
	# ------------------------------------------------------------------------------------------------

	# Find process(es)
	/bin/ps aux | /bin/grep -v grep | /bin/grep --quiet "Waiting to start integrity checks"
}

function self_is_lowest_pid() {
	# ------------------------------------------------------------------------------------------------
	# Function: self_is_lowest_pid()
	# Purpose : Determine if this instance of the script has the lowest PID
	# Returns : 0=Is lowest, 1=Is not lowest
	# ------------------------------------------------------------------------------------------------

	running_script_pids=$(sub_pid="${BASHPID}"; /bin/ps aux | /bin/grep $0 | /bin/grep -v "grep" | /bin/grep -v "${sub_pid}" | /bin/awk '{print $2}')
	if [[ "$(head -n 1 <<< "${running_script_pids}")" == "${my_pid}" ]]; then
		return 0
	else
		return 1
	fi
}

function usb_drive_mounted() {
	# ------------------------------------------------------------------------------------------------
	# Function: usb_drive_mounted()
	# Purpose : Check if any USB drive is currently mounted to a file system matching string
	# Returns : 0=Mounted, 1=Not mounted
	# ------------------------------------------------------------------------------------------------

	local _search_text
	# If we haven't been passed a mount point to look for, assume default
	if (( $# == 0 )); then
		_search_text="USB"
	else
		_search_text="$1"
	fi

	# Command 'df...' returns 0 if any file system is found (mounted), 1 otherwise
	# Limit output of 'df' command to just list the filesystem target ('Mounted on')
	/bin/df --exclude-type=tmpfs | /bin/grep --quiet "${_search_text}"
}

function get_usb_devices() {
	# ------------------------------------------------------------------------------------------------
	# Function: get_usb_devices()
	# Purpose : Get USB block devices matching a passed string. Default to returning all USB devices
	# Output : USB block device IDs if found, empty string if not found
	# Returns : 0=Completed execution, 1=Error
	# ------------------------------------------------------------------------------------------------

	local _search_text
	local _usb_device_paths
	local _usb_device
	local _usb_devices

	# If we haven't been passed a mount point to look for, assume default
	if (( $# == 0 )); then
		_search_text="USB"
	else
		_search_text="$1"
	fi

	# Command 'df...' returns the USB block device ID, or empty string if not found
	if usb_drive_mounted "${_search_text}"; then
		_usb_device_paths=$(/bin/df --exclude-type=tmpfs | /bin/grep "${_search_text}" | /bin/awk '{print $1}')

		for _usb_device in $_usb_device_paths; do
			_usb_devices+=" ${_usb_device##*/}"
		done

		/bin/echo "${_usb_devices}" | /bin/sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
	else
		/bin/echo ""
	fi
}

function get_usb_device_ports() {
	# ------------------------------------------------------------------------------------------------
	# Function: get_usb_device_ports()
	# Purpose : Get a USB device bus and port identifier(s) by search string
	# Output : Bus and Port identifier(s) if found, empty string if not found
	# Returns : 0=Completed execution, 1=Error
	# ------------------------------------------------------------------------------------------------

	local _search_text
	local _usb_device
	local _usb_devices
	local _usb_device_port
	local _usb_device_ports

	_search_text="$1"

	_usb_devices=$(get_usb_devices "${_search_text}")

	for _usb_device in ${_usb_devices}; do
		_usb_device_port=$(/bin/udevadm info -q path -n "${_search_text}" | grep -o "/[0-9]-[0-9]/" 2> /dev/null)

		if (( $? == 0 )); then
			_usb_device_ports+=" ${_usb_device_port//\/}"
		fi
	done
	/bin/echo "${_usb_device_ports}" | /bin/sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
	return 0
}

function get_device_from_share() {
	# ------------------------------------------------------------------------------------------------
	# Function: get_device_from_share()
	# Purpose : Get the device name for a USB disk by remote share name
	# Output : USB device identifier if found, empty string if not found
	# Returns : 0=Completed execution, 1=Error
	# ------------------------------------------------------------------------------------------------

	local _remote_share
	local _disk_info
	local _usb_device_paths

	if (( $# == 0 )); then
		/bin/echo ""
		exit 0
	else
		_remote_share=" $1"
	fi

	_usb_device_paths=$(/bin/df --exclude-type=tmpfs | /bin/grep "/dev/usb" | /bin/awk '{print $1}')

	for _usb_device in ${_usb_device_paths}; do
		_disk_info=$(/usr/syno/bin/synousbdisk -info "${_usb_device##*/}")

		if /bin/grep --quiet "${_remote_share}" <<< "${_disk_info}"; then
			/bin/grep "information:" <<< "${_disk_info}" | awk '{print $2}'
		fi
	done
}

function usb_port_has_mounted_volumes() {
	# ------------------------------------------------------------------------------------------------
	# Function: usb_port_has_mounted_volumes()
	# Purpose : Check if any mounted USB volumes are on a specific USB bus and port
	# Returns : 0=Mounted, 1=Not mounted
	# ------------------------------------------------------------------------------------------------

	local _usb_port
	local _usb_devices

	if (( $# == 0 )); then
		return 1
	else
		_usb_port="$1"
	fi

	_usb_devices=$(get_usb_devices)

	for _usb_device in ${_usb_devices}; do
		local _usb_device_port
		_usb_device_port=$(get_usb_device_ports "${_usb_device}")
		if [[ "${_usb_port}" == "${_usb_device_port}" ]]; then
			return 0
		fi
	done

	return 1
}

function safe_remount_usb_port() {
	# ------------------------------------------------------------------------------------------------
	# Function: safe_remount_usb_port()
	# Purpose : Re-mount a USB port if all devices are unmounted
	# Returns : 0=Found and mounted or detected externally mounted, 1=Could not safely remount
	# ------------------------------------------------------------------------------------------------

	local _usb_port

	_usb_port="$1"

	if [[ -n "$(/usr/syno/bin/lsusb | /bin/grep "${_usb_port}")" ]]; then
		if ! usb_port_has_mounted_volumes "${_usb_port}"; then
			/bin/echo "${_usb_port}" > /sys/bus/usb/drivers/usb/unbind
			/bin/sleep 10
			/bin/echo "${_usb_port}" > /sys/bus/usb/drivers/usb/bind
		else
			return 1
		fi
	else
		return 1
	fi
}

function safe_unmount_usb_device() {
	# ------------------------------------------------------------------------------------------------
	# Function: safe_unmount_usb_device()
	# Purpose : Unmount a USB device
	# Returns : 0=Script ran successfully, 1=Error
	# ------------------------------------------------------------------------------------------------

	local _usb_device
	local _unmount_result

	_usb_device="$1"

	/bin/sync
	/bin/sleep 5
	if usb_drive_mounted "${_usb_device}"; then
		_unmount_result=$(/usr/syno/bin/synousbdisk -umount "${_usb_device}")
		code=$?

		sleep 5;
		if [[ "${code}" == 0 ]]; then
			remove_usb_from_gui "${_usb_device}"
		fi

		/bin/echo "${_unmount_result}"
	fi
}

function remove_usb_from_gui() {
	# ------------------------------------------------------------------------------------------------
	# Function: remove_usb_from_gui()
	# Purpose : Remove a specific USB device(s) from the DSM GUI, if found
	# Returns : 0=Completed execution, 1=Error
	# ------------------------------------------------------------------------------------------------

	local _usb_device
	local _usb_gui_reference

	for _usb_device in "$@"; do
		_usb_gui_reference="${_usb_device%p*}"
		/bin/echo 1 > /sys/block/${_usb_gui_reference}/device/delete;
	done
}

startTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")

if (( $# != 1 )); then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "This script requires one argument."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - This script requires one argument." >> /var/log/synolog/synobackup.log
	exit 1
fi

_backup_wait_seconds=7200 # MAXIMUM time to wait for backup to complete
_integrity_wait_seconds=14400 # MAXIMUM time to wait for all integrity checks

task_string="$1"
task_name="[$1]" # Only used for log entries
my_pid="${BASHPID}"
leader=0
serial_integrity=1

synobackup_config_json=$(/bin/cat /var/packages/HyperBackup/etc/synobackup.conf | /bin/awk '/\[global/{g=1} /\[task_[0-9]+|\[repo_[0-9]+/{f=1} !(g || f) { st = index($0,"="); printf "\"" substr($0,0,st-1) "\": " substr($0,st+1) ","; } g{ printf "{ \"global\": {"; g=0 } f{ s=substr($0, 2, length-2); printf "}, \"" s "\": {"; f=0 }; END { printf "}}"}' | /bin/sed 's/,}/}/g')

if ! /bin/jq type >/dev/null 2>&1 <<< "${synobackup_config_json}"; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "Unable to parse synobackup.conf to JSON. Exiting."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Unable to parse synobackup.conf to JSON. Exiting." >> /var/log/synolog/synobackup.log
	exit 1
fi

my_task_id=$(/bin/jq -r --arg task_string "${task_string}" 'to_entries[] | select(.value.name | strings | contains($task_string)) | .key' <<< "${synobackup_config_json}")
my_task_id_num="${my_task_id##*_}"

if [[ $(/bin/wc -l <<< "${my_task_id}") -gt 1 ]]; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "Task string matched more than one HyperBackup job."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Task string matched more than one HyperBackup job." >> /var/log/synolog/synobackup.log
	exit 1
fi

if [[  $(/bin/wc -l <<< "${my_task_id}") -eq 0 ]]; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "Task string did not match a HyperBackup job."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Task string did not match a HyperBackup job." >> /var/log/synolog/synobackup.log
	exit 1
fi

my_repo_id=repo_$(/bin/jq -r --arg task_id "${my_task_id}" '.[$task_id].repo_id' <<< "${synobackup_config_json}")
my_remote_share=$(/bin/jq -r --arg repo_id "${my_repo_id}" '.[$repo_id].remote_share' <<< "${synobackup_config_json}")
my_device=$(get_device_from_share "${my_remote_share}")

if [[ -z "${my_device}" ]]; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "No USB device found. Exiting."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - No USB device found. Exiting." >> /var/log/synolog/synobackup.log
	exit 1
fi

my_device_port="$(get_usb_device_ports "${my_device}")"

if [[ -z "${my_device_port}" ]]; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "No USB device port found. Exiting."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - No USB device port found. Exiting." >> /var/log/synolog/synobackup.log
	exit 1
fi

#/bin/echo 2 > /dev/ttyS1 # Beep on start

currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
(( debug == 1 )) && /bin/echo "Integrity check task started."
/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Started." >> /var/log/synolog/synobackup.log

## Get my USB bus and port
## Get all mounted shares on my usb port

## Backup - Begin
_end_time=$(( SECONDS + ${_backup_wait_seconds} ))
while usb_drive_mounted "${my_device}" && (( SECONDS < ${_end_time} )); do
	/bin/sleep 20
done

/bin/sleep 20

if usb_drive_mounted "${my_device}"; then
	currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
	(( debug == 1 )) && /bin/echo "Backup task did not eject USB drive before timeout occurred - exiting."
	/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Backup task did not eject USB drive before timeout occurred - exiting." >> /var/log/synolog/synobackup.log
	exit 1
fi
## Backup - End
/bin/sleep 20

## Device mount - Begin
## Remount devices (only one script per usb port, all drives must eject)

## Determine leader by lowest PID. If more than one script instance is scheduled, all should be running at this point.
if self_is_lowest_pid; then
	leader=1
fi

# If leader, wait for all devices on USB port to be unmounted
if [[ "${leader}" == "1" ]]; then
	_end_time=$(( SECONDS + ${_backup_wait_seconds} ))
	
	while usb_port_has_mounted_volumes "${my_device_port}" && (( SECONDS < ${_end_time} )); do
		/bin/sleep 20
	done
	
	/bin/sleep 10
	
	safe_remount_usb_port "${my_device_port}"
fi

## Check if mounted loop
_end_time=$(( SECONDS + ${_backup_wait_seconds} ))
while ! usb_drive_mounted "${my_device}" && (( SECONDS < ${_end_time} )); do
	/bin/sleep 20s
done

## Device mount - End

/bin/sleep $(( ${my_task_id_num} % 60 ))

## Check integrity - Begin
# Sleep for any other running task to run one at a time. Slight race condition possibility here.
# The subshell sleep/test is a very poor substitute for a locking mechanism.
# It can certainly fail, but should be very rare and without significant consequences.
# I also didn't really want to mess with a file based locking mechanism via flock, but it could be done.
if [[ "${serial_integrity}" == "1" ]]; then
	while integrity_process_is_active; do
		/bin/bash -c "/bin/sleep $(( (${my_task_id_num} % 60) * 3 + 30 )); test 'Waiting to start integrity checks'"
	done
fi

/var/packages/HyperBackup/target/bin/detect_monitor --task-id ${my_task_id_num} --trigger --full --guard

# Wait until check is finished, poll every 60 seconds
while integrity_process_is_active "${my_task_id_num}"; do
	/bin/sleep 60
done

## Check integrity - End
currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
(( debug == 1 )) && /bin/echo "Integrity check complete."
(( debug == 1 )) && /bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Integrity check complete." >> /var/log/synolog/synobackup.log

# Sleep a bit more before attempting to unmount the disk
/bin/sleep 10

## Unmount USB device - Begin
## Wait until all integrity tasks are complete.

while integrity_process_is_active || integrity_script_waiting; do
	/bin/sleep 60
done

## Attempt unmount. If already unmounted, no action will be taken.
unmount_result=$(safe_unmount_usb_device "${my_device%p*}")

currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
(( debug == 1 )) && /bin/echo "Disk unmount result: ${unmount_result}"
(( debug == 1 )) && /bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - ${unmount_result}" >> /var/log/synolog/synobackup.log

## Unmount USB device - End

currTime=$(/bin/date +"%Y/%m/%d %H:%M:%S")
(( debug == 1 )) && /bin/echo "Integrity check task finished."
/bin/echo -e "info\t${currTime}\tSYSTEM:\t${task_name} Integrity check task - Finished." >> /var/log/synolog/synobackup.log

## Get results of auto backup (from last lines of log file) - Begin
IFS=''
output=()
NL=$'\n'

while read line
do

    # Compute the seconds since epoch for the start date and time
    t1=$(/bin/date --date="${startTime}" +%s)

    # Date and time in log line (second column)
    dt2=$(/bin/echo "${line}" | cut -d$'\t' -f2)
    # Compute the seconds since epoch for log line date and time
    t2=$(/bin/date --date="${dt2}" +%s)

    # Compute the difference in dates in seconds
    let "tDiff=${t2}-${t1}"

    # Stop reading log lines from before the startTime
    if [[ "${tDiff}" -lt 0 ]]; then
        break
    fi

    text=$(/bin/echo "${line}" | /bin/cut -d$'\t' -f4)
    # Get only lines for this task
	text=$(echo "${text}" | sed 's/\[Local\]//')
    text=$(/bin/echo "${text}" | /bin/grep "\[${task_string}\]")
    # Add date and time
	text=$(/bin/echo "${dt2}  ${text}")

	output+=("${text}")

done <<<$(/bin/tac /var/log/synolog/synobackup.log)

n=${#output[*]}
for (( i = n-1; i >= 0; i-- ))
do

	/bin/echo "${output[i]}"

done
## Get results - End

## Hijack the USB Copy package's "Completed a Task" event to send a notification
## Though it does still show it is from the USB Copy application, the rest of the message is generic enough
## This will send a message at the INFO level. USBCOPYError would be a good tag for the WARN level

if /bin/grep --quiet "USBCOPYFinished=" /usr/syno/etc/notification/notification_filter.settings; then
	while ! self_is_lowest_pid; do
		sleep 120
	done

	/usr/syno/bin/synonotify USBCOPYFinished "{\"%COPY%\":\"${task_string} integrity check\"}"
fi

exit 0