rsync analisys

look the comentary of this gist :D

create basic blocks

$ dd if=/dev/urandom bs=1024 count=1024 of=data/1
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102661 s, 10.2 MB/s
$ dd if=/dev/urandom bs=1024 count=1024 of=data/2
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105779 s, 9.9 MB/s
$ dd if=/dev/urandom bs=1024 count=1024 of=data/3
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915376 s, 11.5 MB/s

build one house.

$ cat data/{3,2,1} > data/house

check

$ ls -l data
total 6144
-rw-rw-r-- 1 postgres postgres 1048576 Aug 15 19:37 1
-rw-rw-r-- 1 postgres postgres 1048576 Aug 15 19:37 2
-rw-rw-r-- 1 postgres postgres 1048576 Aug 15 19:37 3
-rw-rw-r-- 1 postgres postgres 3145728 Aug 15 19:38 house

check md5

$ md5sum data/*
48723344ff2a875a70c6002aef96837e  data/1
ea6d225a1871478994d8f1a192aff3f1  data/2
b4875e90b92215ebbd42c2109b824813  data/3
6e272f9c7b7fd2a5d65e336dbaf00e11  data/house

rsync for the first time.

$ rsync -avP --delete --info=progress2 --stats data/*  pgslave:data/
sending incremental file list
created directory data
1
      1,048,576  16%   96.88MB/s    0:00:00 (xfr#1, to-chk=3/4)
2
      2,097,152  33%   85.60MB/s    0:00:00 (xfr#2, to-chk=2/4)
3
      3,145,728  50%   84.82MB/s    0:00:00 (xfr#3, to-chk=1/4)
a
      6,291,456 100%   59.69MB/s    0:00:00 (xfr#4, to-chk=0/4)

Number of files: 4 (reg: 4)
Number of created files: 4 (reg: 4)
Number of deleted files: 0
Number of regular files transferred: 4
Total file size: 6,291,456 bytes
Total transferred file size: 6,291,456 bytes
Literal data: 6,291,456 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 6,293,258
Total bytes received: 126

sent 6,293,258 bytes  received 126 bytes  12,586,768.00 bytes/sec
total size is 6,291,456  speedup is 1.00

as we can see, it send the three blocks plus the house to the other side.
a total of 6,293,258 bytes were sent through the network.

now lets build a different house with the same blocks.

$ cat data/{1,3,2} > data/house

now lets check md5

$ md5sum data/*
48723344ff2a875a70c6002aef96837e  data/1
ea6d225a1871478994d8f1a192aff3f1  data/2
b4875e90b92215ebbd42c2109b824813  data/3
b86146ad340fef32be90cb1d7918e4cf  data/house

so, blocks 1,2 and 3 are still the same. only the house is new.

now lets rsync it.

$ rsync -avP --delete --info=progress2 --stats data/*  pgslave:data/
sending incremental file list
house
      3,145,728  50%  130.30MB/s    0:00:00 (xfr#1, to-chk=0/4)

Number of files: 4 (reg: 4)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 6,291,456 bytes
Total transferred file size: 3,145,728 bytes
Literal data: 2,224 bytes
Matched data: 3,143,504 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 9,504
Total bytes received: 10,722

sent 9,504 bytes  received 10,722 bytes  40,452.00 bytes/sec
total size is 6,291,456  speedup is 311.06

Now we see that only 9,504 bytes were transfered through the network.

That means that rsync is doing the magic of block transfer. O.o

PS: TODO: test with block files @ 1GB