Skip to content

Instantly share code, notes, and snippets.

@MADscientist314
Created November 2, 2020 17:08
Show Gist options
  • Save MADscientist314/a892e2b50aa7a828f081d88e0d1ff393 to your computer and use it in GitHub Desktop.
Save MADscientist314/a892e2b50aa7a828f081d88e0d1ff393 to your computer and use it in GitHub Desktop.
execute a parallel rsync with xargs for large data transfers
#!/bin/bash
# Parallel rsync script originally designed for rsyncing
# large ata transfers from RAID to RAID for the Aagaard Lab.
# Author: Michael Jochum
# Location: Baylor College of Medicine, Houston, TX, USA
# Contact : [email protected]
# Date : 2 November 2020
##################################
#Step 0: fill this shit out
##################################
#source directory path wehere you are currently located
SRC=$PWD;
#destination [email protected]:/path/to/your/dest/dir
DEST="/path/to/dest/dir"
#number of simultaneous rsync connections
#dont be stupid about this or it will be slower than 1 rsync
CON=10
##################################
#Step 1 change into the source dir
##################################
cd $SRC;
##################################
#Step 1 execute rsync with xargs
##################################
#Step 2 #execute this command
ls $SRC |xargs -n1 -P$CON -I% rsync -Pa % $DEST
@J-Massey
Copy link

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment