Skip to content

Instantly share code, notes, and snippets.

@sloppycoder
Last active January 27, 2020 04:16
Show Gist options
  • Select an option

  • Save sloppycoder/f20f9c030df2a8fcb506964f87c465cd to your computer and use it in GitHub Desktop.

Select an option

Save sloppycoder/f20f9c030df2a8fcb506964f87c465cd to your computer and use it in GitHub Desktop.
large file parallel compression test
centos6 is a directory of a VirtualBox VM, size 8689088000
command for compression
time tar --use-compress-program=pbzip2|lzop -cf centos6_oracle11r2.tar.xxx centos6
time lzrtar centos6
command for decompresison
time pbzip2|bzip2|lzop -cd centos6_oracle11r2.tar.xxx | tar tvf -
time lrzip -o - -d centos6.tar.lrz | tar tvf -
Size Compress Decompress
2930396334 Sep 11 23:37 centos6_oracle11r2.zip 302 58 <-- this is the reference
3708523626 Sep 11 23:56 centos6_oracle11r2.tar.lzo 36 21
2710537316 Sep 11 23:42 centos6_oracle11r2.tar.bz2 213 73 (bzip2: 325)
1842342646 Sep 12 00:42 centos6.tar.lrz 519 750!!
hardware:
Macbook Pro Retina 2012, quad core 2.0G / 16G, original 256G Samsung SSD
versions:
macOS 10.12.6
homebrew 1.3.2, updated 2017-09-12
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
lrzip version 0.631
pbzip2 Parallel BZIP2 v1.1.13 [Dec 18, 2015]
bzip2, a block-sorting file compressor. Version 1.0.6, 6-Sept-2010.
lzop 1.04 LZO library 2.10
related post:
https://askubuntu.com/questions/258202/multi-core-compression-tools
Update 2020-01-27
MSI GF63 RCX quad core i5-9300H, 2.4G, 24G ram, 512G SSD
xz (XZ Utils) 5.2.2
liblzma 5.2.2
l2win.vdi 26,027,753,472 bytes
time ( tar -cf - l2win | xz --threads=4 > l2win_20200127_vbox6.0.16.tar.xz )
openssl md5 l2win_20200127_vbox6.0.16.tar.xz
tar.gz is 7,810,053,656 bytes, 22 mins.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment