Collect restic metrics when using resticker

README.md

Collect restic metrics when using resticker

Restic is a backup software, written in Go, that stands out for the method of backup that is using, called Content Defined Chunking de-duplication. It supports multiple backends and is really easy to use.

I am using resticker, a straight-forward docker container, to run restic in Production.

It allows to schedule the backups with each built-in cron support, and allows me to send notifications for successful/failed backups.

Resticker currently does not support Prometheus metrics, in order to monitor the status of the backups. So I took a manual approach to collect data using the output from official restic commands, and send them to a file, which is picked up by Node Exporter and is send to Prometheus server.

• The script is using jq to parse the json response from restic.
• The script is invoked by crontab.
• docker-compose.override.yml example is provided, on how to send the metric file to Prometheus via Node Exporter.

backup_metrics.sh

#!/bin/bash

# This script will execute restic commands and will pipe the output to jq,
# in order to format it in a prometheus compatible metric.
# Add this script in the host crontab, to be executed every few hours.

HOST=${HOST:-`hostname`}

# Ensure jq package is installed, as it is needed to format
# the restic json response to valid prometheus metric.
dpkg -l jq &> /dev/null
if [ $? -ne 0 ]; then
    sudo apt install jq -y
fi

TEXTFILE_COLLECTOR_DIR=/metrics
# Create directory.
[ -d ${TEXTFILE_COLLECTOR_DIR} ] || mkdir -p ${TEXTFILE_COLLECTOR_DIR}

# Create a temp unique file, that will not be parsed by node exporter.
TEMP_FILE="${TEXTFILE_COLLECTOR_DIR}/restic.prom.$$"
PERM_FILE="${TEXTFILE_COLLECTOR_DIR}/restic.prom"
touch ${TEMP_FILE}

# Note the start time of the script.
START="$(date +%s)"

# Get last backup timestamp
docker exec -i restic-backups restic snapshots --host ${HOST} latest --json | jq -r 'max_by(.time) | .time | sub("[.][0-9]+"; "") | sub("Z"; "+00:00") | def parseDate(date): date | capture("(?<no_tz>.*)(?<tz_sgn>[-+])(?<tz_hr>\\d{2}):(?<tz_min>\\d{2})$") | (.no_tz + "Z" | fromdateiso8601) - (.tz_sgn + "60" | tonumber) * ((.tz_hr | tonumber) * 60 + (.tz_min | tonumber)); parseDate(.) | "restic_last_snapshot_ts \(.)"' >> ${TEMP_FILE}
# Get last backup size in bytes and files count
docker exec -i restic-backups restic stats --host ${HOST} latest --json | jq -r '"restic_stats_total_size_bytes \(.total_size)\nrestic_stats_total_file_count \(.total_file_count)"' >> ${TEMP_FILE}

# Write out metrics to a temporary file.
END="$(date +%s)"
echo "restic_collector_duration_seconds $(($END - $START))" >> ${TEMP_FILE}
echo "restic_collector_last_run_ts ${END}" >> ${TEMP_FILE}

# Rename the temporary file atomically.
# This avoids the node exporter seeing half a file.
# In case a temp file was not created, delete the permanent file,
# to avoid outdated metrics.
mv "${TEMP_FILE}" "${PERM_FILE}" || rm "${PERM_FILE}"

crontab

0 */5 * * * /home/user/backup_metrics.sh

docker-compose.override.yml

version: '3.5'
services:
  nodeexporter:
    volumes:
      - /metrics:/metrics:ro
    command:
      - '--collector.textfile.directory=/metrics'