Skip to content

Instantly share code, notes, and snippets.

@hellvesper
Last active May 3, 2023 09:18
Show Gist options
  • Save hellvesper/36f7196209368126d08c25a466d73f8e to your computer and use it in GitHub Desktop.
Save hellvesper/36f7196209368126d08c25a466d73f8e to your computer and use it in GitHub Desktop.
Replication TrueNAS's apple time machnine snapshots with hetzner's cloud storage box [draft]
#!/bin/sh
LAST_LOCAL_SNAPSHOT=`zfs list -t snapshot | sort -r | egrep '^.+aapltm-[0-9]+' -o -m 1`
LAST_REMOTE_SNAPSHOT=`rclone ls storagebox:/home/ | sort -k 2 -r | grep aapltm -m 1 | xargs | cut -d ' ' -f 2 | cut -d '.' -f 1`
DATASET_PREFIX=`zfs list -t snapshot | grep aapltm -m 1 | cut -d ' ' -f 1 | cut -d '@' -f 1`
LOCAL_SNAPSHOT_POSTFIX=`echo $LAST_LOCAL_SNAPSHOT | cut -d '@' -f 2`
RCLONE_REMOTE="storagebox"
RCLONE_REMOTE_SNAPSHOT_PATH="/home/"
echo "LAST_LOCAL_SNAPSHOT:$LAST_LOCAL_SNAPSHOT"
echo "LAST_REMOTE_SNAPSHOT:$LAST_REMOTE_SNAPSHOT"
echo "DATASET_PREFIX:$DATASET_PREFIX"
echo "LOCAL_SNAPSHOT_POSTFIX:$LOCAL_SNAPSHOT_POSTFIX"
echo "RCLONE_REMOTE:$RCLONE_REMOTE"
echo "RCLONE_REMOTE_SNAPSHOT_PATH:$RCLONE_REMOTE_SNAPSHOT_PATH"
#check if TM snapshot exist
if [ -z "$LAST_LOCAL_SNAPSHOT" ] #test -z <var> => if string has zero lenght
then
echo "Local snapshot not found, is it exist?"
exit 1
fi
if [ -z "$LAST_REMOTE_SNAPSHOT" ]
then
echo "Remote snapshot not found: $LAST_REMOTE_SNAPSHOT"
fi
if [ "$LAST_LOCAL_SNAPSHOT" = "$DATASET_PREFIX@$LAST_REMOTE_SNAPSHOT" ]
then
echo "Local and remote sanphots are equal, backup doesn't needed."
exit 0
else
echo "Local and remote sanphots are NOT equal: $LAST_LOCAL_SNAPSHOT" = "$DATASET_PREFIX@$LAST_REMOTE_SNAPSHOT"
fi
IS_LOCAL_SNAPSHOT_STILL_EXISTS=`zfs list -t snapshot | grep $DATASET_PREFIX"@"$LAST_REMOTE_SNAPSHOT`
if [ -z "$LAST_REMOTE_SNAPSHOT" ] || [ -z "$IS_LOCAL_SNAPSHOT_STILL_EXISTS"]
then
echo "Remote snapshot not found or local copy doesn't exist, prepare for initial backup upload."
UPLOAD_SIZE=`zfs send -nPc $LAST_LOCAL_SNAPSHOT | grep size | cut -d ' ' -f 2`
echo "Uploading " `expr $UPLOAD_SIZE / 1024 / 1024 / 1024` " GiB"
echo "Calculating sha1sum for integrity check, this will take a wile"
LOCAL_CHECKSUM=`sudo zfs send -c $LAST_LOCAL_SNAPSHOT| pv -s $UPLOAD_SIZE -w 80 | sha1`
echo "Local snapshot sha1 sum: $LOCAL_CHECKSUM"
sudo zfs send -c $LAST_LOCAL_SNAPSHOT | rclone -P rcat $RCLONE_REMOTE":"$RCLONE_REMOTE_SNAPSHOT_PATH$LOCAL_SNAPSHOT_POSTFIX".snap" --size $UPLOAD_SIZE
echo "Upload done, prepearing integrity check"
echo "Calculetaing remote sha1 sum"
REMOTE_CHECKSUM=`rclone sha1sum $RCLONE_REMOTE":"$RCLONE_REMOTE_SNAPSHOT_PATH$LOCAL_SNAPSHOT_POSTFIX".snap" | cut -d ' ' -f 1`
if [ "$LOCAL_CHECKSUM" = "$REMOTE_CHECKSUM" ]
then
echo $LOCAL_CHECKSUM "=" $REMOTE_CHECKSUM "ok"
return 0
else
echo $LOCAL_CHECKSUM "!=" $REMOTE_CHECKSUM "checksum not equal!"
return 1
fi
else
echo "Last remote snapshot found:" $LAST_REMOTE_SNAPSHOT "prepare for incremental upload"
UPLOAD_SIZE=`zfs send -nPcI $DATASET_PREFIX"@"$LAST_REMOTE_SNAPSHOT $LAST_LOCAL_SNAPSHOT | grep size | cut -d ' ' -f 2`
echo "Uploading " `expr $UPLOAD_SIZE / 1024 / 1024 / 1024` " GiB"
echo "Calculating local sha1 sum for integrity check, this will take a wile"
LOCAL_CHECKSUM=`sudo zfs send -cI $DATASET_PREFIX"@"$LAST_REMOTE_SNAPSHOT $LAST_LOCAL_SNAPSHOT | pv -s $UPLOAD_SIZE -w 80 | sha1`
echo "Local snapshot sha1 sum: $LOCAL_CHECKSUM"
echo "Uploading snapshot:"
sudo zfs send -cI $DATASET_PREFIX"@"$LAST_REMOTE_SNAPSHOT $LAST_LOCAL_SNAPSHOT | rclone -P rcat $RCLONE_REMOTE":"$RCLONE_REMOTE_SNAPSHOT_PATH$LOCAL_SNAPSHOT_POSTFIX".snap" --size $UPLOAD_SIZE
echo "Upload done, prepearing integrity check"
echo "Calculetaing remote sha1 sum"
REMOTE_CHECKSUM=`rclone sha1sum $RCLONE_REMOTE":"$RCLONE_REMOTE_SNAPSHOT_PATH$LOCAL_SNAPSHOT_POSTFIX".snap" | cut -d ' ' -f 1`
if [ "$LOCAL_CHECKSUM" = "$REMOTE_CHECKSUM" ]
then
echo $LOCAL_CHECKSUM "=" $REMOTE_CHECKSUM "ok"
return 0
else
echo $LOCAL_CHECKSUM "!=" $REMOTE_CHECKSUM "checksum not equal!"
return 1
fi
fi
@hellvesper
Copy link
Author

Examples of work, first snapshot is not incremental:

Remote snapshot not found: 
Local and remote sanphots are NOT equal: Storage/vesper@aapltm-1663738318 = Storage/vesper@
Remote snapshot not found, prepare for initial backup upload
Uploading  377  GiB
Calculating sha1sum for integrity check, this will take a wile
 377GiB 1:59:01 [54.2MiB/s] [================================>] 100%            
Local snapshot sha1 sum: f2e0e1052ed35e53fc3ca92fcaf5c72caf32294e
Transferred:      377.770 GiB / 377.770 GiB, 100%, 7.417 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:  14h47m16.3s
Upload done, prepearing integrity check
Calculetaing remote sha1 sum
f2e0e1052ed35e53fc3ca92fcaf5c72caf32294e = f2e0e1052ed35e53fc3ca92fcaf5c72caf32294e ok

Incremental backup:

Local and remote sanphots are NOT equal: Storage/vesper@aapltm-1663807395 = Storage/vesper@aapltm-1663738318
Last remote snapshot found: aapltm-1663738318 prepare for incremental upload
Uploading  4  GiB
Calculating local sha1 sum for integrity check, this will take a wile
4.77GiB 0:01:57 [41.6MiB/s] [================================> ] 99%            
Local snapshot sha1 sum: 70885ba85454e114d5ee767a9ef12034073e4e36
Uploading snapshot:
Transferred:        4.766 GiB / 4.766 GiB, 100%, 7.853 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:     10m31.4s
Upload done, prepearing integrity check
Calculetaing remote sha1 sum
70885ba85454e114d5ee767a9ef12034073e4e36 = 70885ba85454e114d5ee767a9ef12034073e4e36 ok

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment