Last active
May 30, 2019 11:41
-
-
Save rustam18/c9c800c096ed77cbb70c7235a75bb179 to your computer and use it in GitHub Desktop.
Concurrent sync with parallel threads increase your sync speed by 4 times
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| cat /home/rusti/<dir_list>.txt | parallel --will-cite -j4 time rsync -ahrzp user@<source>:<source_dir> <dest_dir> | |
| The fastest remote directory rsync over ssh using GNU utility parallel which will run concurrent threads at the same time. | |
| a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files. | |
| h: human-readable output numbers in a human-readable format | |
| r: recursive | |
| z: compress during transfer | |
| p: progress | |
| -will-cite -j4: four threads, it should not be more than the 1.5 * number of cpu chores you have on your system. | |
| time: gives you the transfer time |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment