Skip to content

Instantly share code, notes, and snippets.

@timabell
Last active September 1, 2024 21:22
Show Gist options
  • Save timabell/f70f34f8933b2abaf42789f8afdbd7d5 to your computer and use it in GitHub Desktop.
Save timabell/f70f34f8933b2abaf42789f8afdbd7d5 to your computer and use it in GitHub Desktop.
hashdeep orchestrator for data integrity checks in home folder
#!/bin/sh -v
cd ~
# in docs subdirectory so it is sync'd with syncthing
hash_file="Documents/hashdeep-checksums.txt"
mv "$hash_file" "$hash_file.bak"
backup_paths=(Music Downloads Documents Pictures Phone Dropbox repo oneplus9-home)
time hashdeep -c md5 -of -r -l $folders | tee "$hash_file"
#!/bin/bash
set -e
# usage:
# verify.sh [home folder / backup root]
# e.g.
# ./verify.sh # verify home folder
# ./verify.sh /media/tim/backup/backintime/any/tim/1/20240830-234307-766/backup/home/tim # verify backup folder
# pre-requisites:
# https://github.com/timabell/paths2html binary must be on the path
# in docs subdirectory so it is sync'd with syncthing
hash_file="Documents/hashdeep-checksums.txt"
if [ $# -lt 1 ]; then
backuproot="$HOME"
backup_paths=(Music Downloads Documents Pictures Phone Dropbox repo oneplus9-home)
else
backuproot="$1"
backup_paths=.
fi
tmp_dir=$(mktemp -d -t verify-XXXXXXXXXX)
output_file="$tmp_dir/hashdeep-checksums-verification.txt"
echo "backup root: $backuproot"
echo "temp folder: $tmp_dir"
echo "output_file: $output_file"
cd "$backuproot"
# re-hash everything to a temporary file
time hashdeep -k "$hash_file" -rle -of $backup_paths -avv | tee "$output_file"
# Odd terminology from md5deep verify:
# - "Known file not used" means there was no match for this hash, i.e. deleted file
# - "No match" means new/modified file
grep "Known file not used" "$output_file" | sed 's/:.*//'| sort | paths2html | tee "$missing_file" && xdg-open "$missing_file"
@timabell
Copy link
Author

timabell commented Sep 1, 2024

bug:

hashdeep verify gets confused by multiple identical copies of the same file and ends up reporting a bunch of moves and "known file not used" that are incorrect

╭─tim@fox /media/tim/backup/backintime/any/tim/1/20240830-234307-766/backup/home/tim 
╰─$ ag "and shadows" Documents/hashdeep-checksums.txt
406:19203681,edeb756eb28d54cf1cfb9313982cc49f,Music/mp3/Moving Shadow - Omni Trio 01.2 The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3
590:19203681,edeb756eb28d54cf1cfb9313982cc49f,Music/mp3/2001.2 Omni Trio  - The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3
480758:19203681,edeb756eb28d54cf1cfb9313982cc49f,oneplus9-home/Music/Moving Shadow - Omni Trio 01.2 The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3
481233:19203681,edeb756eb28d54cf1cfb9313982cc49f,oneplus9-home/Music/2001.2 Omni Trio  - The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3
╭─tim@fox /media/tim/backup/backintime/any/tim/1/20240830-234307-766/backup/home/tim 
╰─$ ag "and shadows" /tmp/verify-BTYozi1oZg/hashdeep-checksums-verification.txt
46730:./oneplus9-home/Music/Moving Shadow - Omni Trio 01.2 The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3: Moved from Music/mp3/Moving Shadow - Omni Trio 01.2 The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3
46820:./oneplus9-home/Music/2001.2 Omni Trio  - The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3: Moved from Music/mp3/Moving Shadow - Omni Trio 01.2 The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3
591213:./Music/mp3/Moving Shadow - Omni Trio 01.2 The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3: Moved from Music/mp3/Moving Shadow - Omni Trio 01.2 The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3
591216:./Music/mp3/2001.2 Omni Trio  - The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3: Moved from Music/mp3/Moving Shadow - Omni Trio 01.2 The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3
614655:Music/mp3/2001.2 Omni Trio  - The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3: Known file not used
818641:oneplus9-home/Music/Moving Shadow - Omni Trio 01.2 The Angels and Shadows Project/Omni Trio, 01 - The Angels & Shadows Project (01.2 Mix).mp3: Known file not used

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment