Skip to content

Instantly share code, notes, and snippets.

@jordansissel

jordansissel/data.md

Created July 3, 2012 23:36
Show Gist options
  • Save jordansissel/3044155 to your computer and use it in GitHub Desktop.
Save jordansissel/3044155 to your computer and use it in GitHub Desktop.
Download ZIP
Compression on large JSON file
Raw
data.md

The Data

Compression

  • Original: 708MB
  • xz -3: 70MB, 5:41.02 (2.07 mb/sec - 10:1 ratio)
  • bzip2 -3: 74MB, 4:39.11 (2.53 mb/sec - 9.5:1 ratio)
  • gzip -3: 103MB: 0:15.15 (46.73 mb/sec - 6.87:1 ratio)
  • lzop -3: 146MB, 0:06.53 (108.42 mb/sec - 4.85:1 ratio)

Decompression

(value is seconds, lower is faster)

  • bzip2 - 39.4
  • xz - 14.97
  • gzip - 7.98
  • lzop - 2.37

The Test

echo bzip2 gzip xz lzop | xargs -tn1 sh -c 'time $1 -3c data.json > data.json.$1' -

produces 'data.json.{compression_type}' files

@mdumitrean
Copy link

How about brotli?

@muthukumarse
Copy link

xz is best from my usage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment