Skip to content

Instantly share code, notes, and snippets.

@Xorlev

Xorlev/gzip chain

Created August 30, 2013 02:54
Show Gist options
  • Save Xorlev/6385885 to your computer and use it in GitHub Desktop.
Save Xorlev/6385885 to your computer and use it in GitHub Desktop.
Download ZIP
Fast GZIP copy, credit the smart guys at Tumblr. You can use this to replicate data from one machine to another, and without decompressing send the data to the next server in the chain until the terminal link in the chain. pigz is a parallel gzip. With this you can copy data at (almost) gigabit line speed. Great for seeding read slaves.
Raw
gzip chain
On the last machine in the chain:
nc -l 1234 | pigz -d | tar xvf -
On each intermediate machine, you use a fifo to copy the data to the next machine as well as decompressing.
mkfifo myfifo
nc $NEXT_SERVER 1234 <myfifo &
nc -l 1234 | tee myfifo | pigz -d | tar xvf -
On the first machine:
tar cv $SOME_DIR | pigz | nc $SERVER 1234
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment