Last active
February 22, 2023 20:22
-
-
Save luis-puhl/394d5741ed5776fd06781d2343a49e96 to your computer and use it in GitHub Desktop.
Using Gzip, multicore, SSH-tunnel to transfer files faster
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # the same, no tunnel. | |
| ssh remote "tar -c /some/stuff | pv | pigz" | pigz -d | tar x | |
| ### Let's start by going to the remote with SSH, also leaving a SSH-Tunnel | |
| # you@local:~ $ | |
| ssh remote -L 31000:127.0.0.1:31000 | |
| ### Then we: | |
| ### (-c) create a TAR from ('/some/stuff') dir, | |
| ### PV to se how much has been read | |
| ### PGIZ to gzip using parallel cores | |
| ### start a NETCAT | |
| ### (-l) listening on | |
| ### (-s 127.0.0.1)(-p 31000) the tunnel | |
| ### (-v) with details | |
| # yourself@remote:~ $ | |
| tar -c /some/stuff | pv | pigz | nc -lvnp 31000 127.0.0.1 | |
| ### On another tab, term, whatever, we get the files from the remote: | |
| ### start a NETCAT | |
| ### (-w 2) with a 2 second tolerance | |
| ### (127.0.0.1 31000) to the tunnel | |
| ### (-v) with details | |
| ### PGIZ to (-d) de-gzip using parallel cores | |
| ### (-x) extract from the TAR and dump-it on the current directory, | |
| # you@local:~ $ | |
| nc -v -w 2 127.0.0.1 31000 | pigz -d | tar x |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment