Replication TrueNAS's apple time machnine snapshots with hetzner's cloud storage box [draft]

hetzner.sh

#!/bin/sh
LAST_LOCAL_SNAPSHOT=`zfs list -t snapshot | sort -r | egrep '^.+aapltm-[0-9]+' -o -m 1`
LAST_REMOTE_SNAPSHOT=`rclone ls storagebox:/home/ | sort -k 2 -r | grep aapltm -m 1 | xargs | cut -d ' ' -f 2 | cut -d '.' -f 1`
DATASET_PREFIX=`zfs list -t snapshot | grep aapltm -m 1 | cut -d ' ' -f 1 | cut -d '@' -f 1`
LOCAL_SNAPSHOT_POSTFIX=`echo $LAST_LOCAL_SNAPSHOT | cut -d '@' -f 2`
RCLONE_REMOTE="storagebox"
RCLONE_REMOTE_SNAPSHOT_PATH="/home/"

echo	"LAST_LOCAL_SNAPSHOT:$LAST_LOCAL_SNAPSHOT"
echo	"LAST_REMOTE_SNAPSHOT:$LAST_REMOTE_SNAPSHOT"
echo	"DATASET_PREFIX:$DATASET_PREFIX"
echo	"LOCAL_SNAPSHOT_POSTFIX:$LOCAL_SNAPSHOT_POSTFIX"
echo	"RCLONE_REMOTE:$RCLONE_REMOTE"
echo	"RCLONE_REMOTE_SNAPSHOT_PATH:$RCLONE_REMOTE_SNAPSHOT_PATH"

#check if TM snapshot exist
if [ -z "$LAST_LOCAL_SNAPSHOT" ] #test -z <var> => if string has zero lenght
	then
	echo "Local snapshot not found, is it exist?"
	exit 1
fi

if [ -z "$LAST_REMOTE_SNAPSHOT" ]
then
	echo "Remote snapshot not found: $LAST_REMOTE_SNAPSHOT"
fi

if [ "$LAST_LOCAL_SNAPSHOT" = "$DATASET_PREFIX@$LAST_REMOTE_SNAPSHOT" ]
then
	echo "Local and remote sanphots are equal, backup doesn't needed."
	exit 0
else
	echo "Local and remote sanphots are NOT equal: $LAST_LOCAL_SNAPSHOT" = "$DATASET_PREFIX@$LAST_REMOTE_SNAPSHOT"
fi

IS_LOCAL_SNAPSHOT_STILL_EXISTS=`zfs list -t snapshot | grep $DATASET_PREFIX"@"$LAST_REMOTE_SNAPSHOT`

if [ -z "$LAST_REMOTE_SNAPSHOT" ] || [ -z "$IS_LOCAL_SNAPSHOT_STILL_EXISTS"]
then
 echo "Remote snapshot not found or local copy doesn't exist, prepare for initial backup upload."
 UPLOAD_SIZE=`zfs send -nPc $LAST_LOCAL_SNAPSHOT | grep size | cut -d '	' -f 2`
 echo "Uploading " `expr $UPLOAD_SIZE / 1024 / 1024 / 1024` " GiB"
 echo "Calculating sha1sum for integrity check, this will take a wile"
 LOCAL_CHECKSUM=`sudo zfs send -c $LAST_LOCAL_SNAPSHOT| pv -s $UPLOAD_SIZE -w 80 | sha1`
 echo "Local snapshot sha1 sum: $LOCAL_CHECKSUM"
 sudo zfs send -c $LAST_LOCAL_SNAPSHOT | rclone -P rcat $RCLONE_REMOTE":"$RCLONE_REMOTE_SNAPSHOT_PATH$LOCAL_SNAPSHOT_POSTFIX".snap" --size $UPLOAD_SIZE
 echo "Upload done, prepearing integrity check"
 echo "Calculetaing remote sha1 sum"
 REMOTE_CHECKSUM=`rclone sha1sum $RCLONE_REMOTE":"$RCLONE_REMOTE_SNAPSHOT_PATH$LOCAL_SNAPSHOT_POSTFIX".snap" | cut -d ' ' -f 1`

 if [ "$LOCAL_CHECKSUM" = "$REMOTE_CHECKSUM" ]
 then
  echo $LOCAL_CHECKSUM "=" $REMOTE_CHECKSUM "ok"
  return 0
 else
  echo $LOCAL_CHECKSUM "!=" $REMOTE_CHECKSUM "checksum not equal!"
  return 1
 fi
else
 echo "Last remote snapshot found:" $LAST_REMOTE_SNAPSHOT "prepare for incremental upload"
 UPLOAD_SIZE=`zfs send -nPcI $DATASET_PREFIX"@"$LAST_REMOTE_SNAPSHOT $LAST_LOCAL_SNAPSHOT | grep size | cut -d '	' -f 2`
 echo "Uploading " `expr $UPLOAD_SIZE / 1024 / 1024 / 1024` " GiB"
 echo "Calculating local sha1 sum for integrity check, this will take a wile"
 LOCAL_CHECKSUM=`sudo zfs send -cI $DATASET_PREFIX"@"$LAST_REMOTE_SNAPSHOT $LAST_LOCAL_SNAPSHOT | pv -s $UPLOAD_SIZE -w 80 | sha1`
 echo "Local snapshot sha1 sum: $LOCAL_CHECKSUM"
 echo "Uploading snapshot:"
 sudo zfs send -cI $DATASET_PREFIX"@"$LAST_REMOTE_SNAPSHOT $LAST_LOCAL_SNAPSHOT | rclone -P rcat $RCLONE_REMOTE":"$RCLONE_REMOTE_SNAPSHOT_PATH$LOCAL_SNAPSHOT_POSTFIX".snap" --size $UPLOAD_SIZE
 echo "Upload done, prepearing integrity check"
 echo "Calculetaing remote sha1 sum"
 REMOTE_CHECKSUM=`rclone sha1sum $RCLONE_REMOTE":"$RCLONE_REMOTE_SNAPSHOT_PATH$LOCAL_SNAPSHOT_POSTFIX".snap" | cut -d ' ' -f 1`

 if [ "$LOCAL_CHECKSUM" = "$REMOTE_CHECKSUM" ]
 then
  echo $LOCAL_CHECKSUM "=" $REMOTE_CHECKSUM "ok"
  return 0
 else
  echo $LOCAL_CHECKSUM "!=" $REMOTE_CHECKSUM "checksum not equal!"
  return 1
 fi
fi

TrueNAS make snapshots of every successful TimeMachine backup. The idea is to replicate those snapshots to cloud as a backup of backup but in efficient way, only an incremental changes. But an initial backup will be a full TM copy anyway.

• Script do check if remote snapshot exist, if it doesn't then produce an initial backup of hundreds GiB, depends on you Mac storage capacity and space used.
• If any snapshot already exists on a remote storage, then script produce an incremental backup between your last local snapshot and last remote cloud snapshot, thx to ZFS it possible. And all backups also compressed with thx to another ZFS feature, which will save (in my case) approx 1.5x of space on your NAS and Cloud. An incremental snapshots will be few GiB, 2-5 GiB in my case.

At the end of snapshot upload script check sha1 checksum with local and remote snapshot copy.

This is a draft, there is no some fault checks, and still no behaviour in case where checksum is wrong, it should delete fault snapshot and reupload it.

Note:
To work properly you should add sudo with No Password for your truenas user. To do this:
sudo nano /usr/local/etc/sudoers locate string like your_username ALL=(ALL) ALL and replace with your_username ALL=(ALL) NOPASSWD: ALL. But this change will lose when you update your NAS.
Another important thing is configure your Hetzner Storage Box ssh login with ssh keys https://docs.hetzner.com/robot/storage-box/backup-space-ssh-keys

Examples of work, first snapshot is not incremental:

Remote snapshot not found: 
Local and remote sanphots are NOT equal: Storage/vesper@aapltm-1663738318 = Storage/vesper@
Remote snapshot not found, prepare for initial backup upload
Uploading  377  GiB
Calculating sha1sum for integrity check, this will take a wile
 377GiB 1:59:01 [54.2MiB/s] [================================>] 100%            
Local snapshot sha1 sum: f2e0e1052ed35e53fc3ca92fcaf5c72caf32294e
Transferred:      377.770 GiB / 377.770 GiB, 100%, 7.417 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:  14h47m16.3s
Upload done, prepearing integrity check
Calculetaing remote sha1 sum
f2e0e1052ed35e53fc3ca92fcaf5c72caf32294e = f2e0e1052ed35e53fc3ca92fcaf5c72caf32294e ok

Incremental backup:

Local and remote sanphots are NOT equal: Storage/vesper@aapltm-1663807395 = Storage/vesper@aapltm-1663738318
Last remote snapshot found: aapltm-1663738318 prepare for incremental upload
Uploading  4  GiB
Calculating local sha1 sum for integrity check, this will take a wile
4.77GiB 0:01:57 [41.6MiB/s] [================================> ] 99%            
Local snapshot sha1 sum: 70885ba85454e114d5ee767a9ef12034073e4e36
Uploading snapshot:
Transferred:        4.766 GiB / 4.766 GiB, 100%, 7.853 MiB/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:     10m31.4s
Upload done, prepearing integrity check
Calculetaing remote sha1 sum
70885ba85454e114d5ee767a9ef12034073e4e36 = 70885ba85454e114d5ee767a9ef12034073e4e36 ok