Bash implementation of a killswitch to check that backups are up-to-date.

Bash Killswitch.md

Bash Killswitch

This is a bash implementation of a killswitch, which will monitor some action and raise an alert when the action fails or stops being performed.

This particular implementation runs an hourly backup script, and runs the killswitch daily at 4 AM to raise an alert if backups from the previous 24 hours failed.

The cron job runs an hourly backup command:

run-backup.sh && touch killswitch

This will touch the status file alive only if the run-backup.sh script succeeds (that's the && notation).

If the backup script fails, the killswitch status file will not be touched, and the killswitch will be activated.

The cron job runs a daily killswitch script:

killswitch.sh

this runs each day at 4 AM.

crontab

# Run the backup command at the top of each hour
0 * * * * run-backup.sh && touch alive

# Run the dead man's switch at 4:00 AM each day
0 4 * * * killswitch.sh

killswitch.sh

#!/bin/bash
#
# This script activates an alert
# if a file is older than TIME_LIMIT.
# This is a killswitch.
# 
# On Mac, use gstat instead of stat
STAT="stat"

# Time limit in seconds
TIME_LIMIT=$((24*60*60))

# State file for updating last touch
STATE_FILE=alive

# Last access time of the state file (in epoch)
last_touch=$($STAT -c %Y $STATE_FILE)

# Current time (in epoch)
current_time=$(date +%s)

# How much time is remaining before the switch fires
timeleft=$((current_time - last_touch))

if [ $timeleft -gt $TIME_LIMIT ]; then
  echo " ~ ~ ~ KILLSWITCH ENGAGED: JOB FAILED ~ ~ ~ "
fi