copy big data amounts quickly from remote machines

fast_copy_big_data.txt

###  install qpress from here: http://www.quicklz.com/
# wget http://www.quicklz.com/qpress-11-linux-x64.tar 
# tar xvf qpress-11-linux-x64.tar 

## switch nc to traditional on ubuntu: 
>sudo update-alternatives --config nc
There are 2 choices for the alternative nc (providing /bin/nc).

  Selection    Path                 Priority   Status
------------------------------------------------------------
* 0            /bin/nc.openbsd       50        auto mode
  1            /bin/nc.openbsd       50        manual mode
  2            /bin/nc.traditional   10        manual mode <--- pick this!

## pv is to show progress, could be left out
## 8888 is just an open port, could be any other port

# on the server to copy from .... run in a screen/tmux session
(cd /var/lib/mysql/ && tar -cf - .) | qpress -voi some.tar | pv -b | nc -q 10 -l -p 8888

# on the server to copy to. .... also run in a screen/tmux session
nc -w 10 your.server.ip.or.domain 8888  | qpress -dvio | tar -xpf -