zoispag · August 16, 2021 09:57 · May 30, 2020 · May 27, 2020
diff --git a/README.md b/README.md
@@ -5,7 +5,7 @@ It supports multiple backends and is really easy to use.
 
 I am using [resticker](https://github.com/djmaze/resticker), a straight-forward [docker](https://hub.docker.com/r/mazzolino/restic/) container, to run restic in Production.
 
-It allows to schedule the backups with each built-in cron support, and allows me to send notifications for successful/failed backups.
+It allows to schedule the backups with its built-in cron support, and allows me to send notifications for successful/failed backups.
 
 Resticker [currently does not support](https://github.com/djmaze/resticker/issues/26) [Prometheus metrics](https://prometheus.io/), in order to monitor the status of the backups.
 So I took a manual approach to collect data using the output from official restic commands, and send them to a file, which is picked up by [Node Exporter](https://github.com/prometheus/node_exporter) and is send to Prometheus server.

diff --git a/README.md b/README.md
@@ -0,0 +1,15 @@
+# Collect restic metrics when using resticker
+
+[Restic](https://restic.net/) is a backup software, written in Go, that stands out for the method of backup that is using, called [Content Defined Chunking](https://restic.net/blog/2015-09-12/restic-foundation1-cdc) de-duplication.
+It supports multiple backends and is really easy to use.
+
+I am using [resticker](https://github.com/djmaze/resticker), a straight-forward [docker](https://hub.docker.com/r/mazzolino/restic/) container, to run restic in Production.
+
+It allows to schedule the backups with each built-in cron support, and allows me to send notifications for successful/failed backups.
+
+Resticker [currently does not support](https://github.com/djmaze/resticker/issues/26) [Prometheus metrics](https://prometheus.io/), in order to monitor the status of the backups.
+So I took a manual approach to collect data using the output from official restic commands, and send them to a file, which is picked up by [Node Exporter](https://github.com/prometheus/node_exporter) and is send to Prometheus server.
+
+- The script is using [jq](https://stedolan.github.io/jq/) to parse the json response from restic.
+- The script is invoked by crontab.
+- `docker-compose.override.yml` example is provided, on how to send the metric file to Prometheus via Node Exporter.
diff --git a/backup_metrics.sh b/backup_metrics.sh
@@ -0,0 +1,42 @@
+#!/bin/bash
+
+# This script will execute restic commands and will pipe the output to jq,
+# in order to format it in a prometheus compatible metric.
+# Add this script in the host crontab, to be executed every few hours.
+
+HOST=${HOST:-`hostname`}
+
+# Ensure jq package is installed, as it is needed to format
+# the restic json response to valid prometheus metric.
+dpkg -l jq &> /dev/null
+if [ $? -ne 0 ]; then
+    sudo apt install jq -y
+fi
+
+TEXTFILE_COLLECTOR_DIR=/metrics
+# Create directory.
+[ -d ${TEXTFILE_COLLECTOR_DIR} ] || mkdir -p ${TEXTFILE_COLLECTOR_DIR}
+
+# Create a temp unique file, that will not be parsed by node exporter.
+TEMP_FILE="${TEXTFILE_COLLECTOR_DIR}/restic.prom.$$"
+PERM_FILE="${TEXTFILE_COLLECTOR_DIR}/restic.prom"
+touch ${TEMP_FILE}
+
+# Note the start time of the script.
+START="$(date +%s)"
+
+# Get last backup timestamp
+docker exec -i restic-backups restic snapshots --host ${HOST} latest --json | jq -r 'max_by(.time) | .time | sub("[.][0-9]+"; "") | sub("Z"; "+00:00") | def parseDate(date): date | capture("(?<no_tz>.*)(?<tz_sgn>[-+])(?<tz_hr>\\d{2}):(?<tz_min>\\d{2})$") | (.no_tz + "Z" | fromdateiso8601) - (.tz_sgn + "60" | tonumber) * ((.tz_hr | tonumber) * 60 + (.tz_min | tonumber)); parseDate(.) | "restic_last_snapshot_ts \(.)"' >> ${TEMP_FILE}
+# Get last backup size in bytes and files count
+docker exec -i restic-backups restic stats --host ${HOST} latest --json | jq -r '"restic_stats_total_size_bytes \(.total_size)\nrestic_stats_total_file_count \(.total_file_count)"' >> ${TEMP_FILE}
+
+# Write out metrics to a temporary file.
+END="$(date +%s)"
+echo "restic_collector_duration_seconds $(($END - $START))" >> ${TEMP_FILE}
+echo "restic_collector_last_run_ts ${END}" >> ${TEMP_FILE}
+
+# Rename the temporary file atomically.
+# This avoids the node exporter seeing half a file.
+# In case a temp file was not created, delete the permanent file,
+# to avoid outdated metrics.
+mv "${TEMP_FILE}" "${PERM_FILE}" || rm "${PERM_FILE}"
diff --git a/crontab b/crontab
@@ -0,0 +1 @@
+0 */5 * * * /home/user/backup_metrics.sh
diff --git a/docker-compose.override.yml b/docker-compose.override.yml
@@ -0,0 +1,7 @@
+version: '3.5'
+services:
+  nodeexporter:
+    volumes:
+      - /metrics:/metrics:ro
+    command:
+      - '--collector.textfile.directory=/metrics'