Skip to content

Instantly share code, notes, and snippets.

@dch
Last active July 18, 2025 07:39
Show Gist options
  • Save dch/b5df789dfb7ee1e8b8afa6263605efc7 to your computer and use it in GitHub Desktop.
Save dch/b5df789dfb7ee1e8b8afa6263605efc7 to your computer and use it in GitHub Desktop.
tarsnap hacky parallel restore script
#!/bin/sh
# recover all files in parallel from the most recent archive
# MIT license
# https://git.io/vdrbG
# "works on my machine"
# lots of assumptions notably path length (strip-component)
# get the latest archive as our names can be sorted by time
ARCHIVE=`tarsnap --keyfile /tmp/tarsnap.key --list-archives | sort | tail -1`
# order the archives by descending size
FILES=`tarsnap --keyfile /tmp/tarsnap.key -tvf ${ARCHIVE} | cut -w -f 5,9 | sort -rn | cut -w -f 2`
# spawn 10 invocations in parallel (use -P 0 for unlimited)
echo $FILES | xargs -P 10 -n 1 -t \
time tarsnap \
--retry-forever \
-S \
--strip-components 6 \
--print-stats \
--humanize-numbers \
--keyfile /tmp/tarsnap.key \
--chroot \
-xv \
-f ${ARCHIVE}
# profit
@c0dev0id
Copy link

Why all this cut and sort in the FILES line?
I'm currently pretty happy with my simplified version:

mkdir -p ~/tarsnap_restore
tarsnap -tf "${ARCHIVE}" \
	| xargs -P 20 -n 20 -t \
	| tarsnap \
		--humanize-numbers \
		--resume-extract \
		--chroot \
		-xf "${ARCHIVE}" \
		-C ~/tarsnap_restore \
		 -- 

I omitted -v, so no cut is necessary to cut out the path.
I added --resume-extract to skip already extracted files.
I added -C ... to extract to a specific dir and not to $PWD.
-P 20 -n 20 is reasonable for my case.
Adding " -- " to the end helps with filenames that contain " -" or " --" that might otherwise be interpreted as tarsnap option.

Other options like -S, --retry-forever, --strip-components 6, seem specific to the authors use-case. So I omitted them for mine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment