Compression on large JSON file

data.md

The Data

Compression

• Original: 708MB
• xz -3: 70MB, 5:41.02 (2.07 mb/sec - 10:1 ratio)
• bzip2 -3: 74MB, 4:39.11 (2.53 mb/sec - 9.5:1 ratio)
• gzip -3: 103MB: 0:15.15 (46.73 mb/sec - 6.87:1 ratio)
• lzop -3: 146MB, 0:06.53 (108.42 mb/sec - 4.85:1 ratio)

Decompression

(value is seconds, lower is faster)

• bzip2 - 39.4
• xz - 14.97
• gzip - 7.98
• lzop - 2.37

The Test

echo bzip2 gzip xz lzop | xargs -tn1 sh -c 'time $1 -3c data.json > data.json.$1' -

produces 'data.json.{compression_type}' files