coco-1-vs-2

Rougly 40% more compression on the data:
-rw-r--r-- 1 proycon proycon 221M Oct  9 23:24 europarl7-en.txt
-rw-r--r-- 1 proycon proycon  96M Oct  9 23:26 v1/europarl7-en.colibri.dat
-rw-r--r-- 1 proycon proycon  59M Oct  9 23:26 v2/europarl7-en.colibri.dat

Comparison with other compressions on plaintext (default settings):
-rw-r--r-- 1 proycon proycon  71M Oct 10 00:11 europarl7-en.txt.gz
-rw-r--r-- 1 proycon proycon  71M Oct 10 00:14 europarl7-en.txt.zip
-rw-r--r-- 1 proycon proycon  49M Oct 10 00:12 europarl7-en.txt.bz2

Decoding speed is good:
v1: colibri-classdecode -f europarl7-en.colibri.dat -c europarl7-en.colibri.cls >  6.96s user 2.49s system 99% cpu 9.459 total
v2: colibri-classdecode -f europarl7-en.colibri.dat -c europarl7-en.colibri.cls >  6.36s user 2.46s system 99% cpu 8.822 total

Compression effect still good on unindexed pattern models:
-rw-r--r-- 1 proycon proycon 90M Oct  9 23:40 v1/europarl7-en.colibri.t2.u.colibri.patternmodel
-rw-r--r-- 1 proycon proycon 69M Oct  9 23:42 v2/europarl7-en.colibri.t2.u.colibri.patternmodel

Memory usage unindexed pattern models:
 v1: 
  Total key bytesize (patterns): 115881796 bytes (110 MB)
  Total value bytesize (counts/index): 29291020 bytes (27 MB)
  Mean key bytesize: 15.8249
  Total bytesize (without overhead): 145172816 bytes (138 MB)
 v2:
  Total key bytesize (patterns): 94205468 bytes (89 MB)
  Total value bytesize (counts/index): 29291020 bytes (27 MB)
  Mean key bytesize: 12.8648
  Total bytesize (without overhead): 123496488 bytes (117 MB)

 Note: Actual reserved memory is much higher (due to map overhead) and show less difference:
  v1: 0.0   0.3  662M  648M 141296 proycon     0 S   0:14.73     0     0 colibri-patternmodeller -i europarl7-en.colibri.t2.u.colibri.patternmodel -c europarl7-en.colibri.cls -Q 
  v2: 0.0   0.2  652M  639M  1534 proycon      0 S   0:13.90     0     0 colibri-patternmodeller -i europarl7-en.colibri.t2.u.colibri.patternmodel -c europarl7-en.colibri.cls -Q

Loading times unindexed models:: 
 v1: colibri-patternmodeller -i europarl7-en.colibri.t2.u.colibri.patternmodel -V  12.96s user 5.06s system 99% cpu 18.029 total
 v2: colibri-patternmodeller -i europarl7-en.colibri.t2.u.colibri.patternmodel -V  11.23s user 4.69s system 100% cpu 15.914 total
 
Disk usage indexed models (little impact of v2 encoding, as index takes up far more space)
-rw-r--r-- 1 proycon proycon 835M Oct 10 00:11 v1/europarl7-en.colibri.t2.indexed.colibri.patternmodel
-rw-r--r-- 1 proycon proycon 815M Oct 10 00:13 v2/europarl7-en.colibri.t2.indexed.colibri.patternmodel

Training a pattern model (including serialisation of result model) is a bit faster on v2:
v1 unindexed: colibri-patternmodeller -f europarl7-en.colibri.dat -o  -t 2 -u -l 4  157.32s user 4.83s system 100% cpu 2:42.13 total
v2 unindexed: colibri-patternmodeller -f europarl7-en.colibri.dat -o  -t 2 -u -l 4  137.67s user 5.13s system 99% cpu 2:22.82 total

v1 indexed: colibri-patternmodeller -f europarl7-en.colibri.dat -o  -t 2 -l 4  211.05s user 9.16s system 100% cpu 3:40.18 total
v2 indexed: colibri-patternmodeller -f europarl7-en.colibri.dat -o  -t 2 -l 4  177.68s user 7.01s system 99% cpu 3:05.43 total

coco-v1-v2-patternmodel

(Tested on Europarl v7, English, 41,960,603 words)

====== 0 - Building class encoding======
v1 ---> time: 15.992 s res: 23.9102 MB peak: 25.9414 MB
v2 ---> time: 12.5389 s res: 23.9023 MB peak: 25.7734 MB

====== 0 - Encoding corpus======
v1 ---> time: 25.8001 s res: 0.109375 MB peak: 25.9414 MB
v2 ---> time: 17.7399 s res: 0.113281 MB peak: 25.7734 MB

====== 3 - Loading corpus data (IndexedCorpus)======
v1 ---> time: 12.9537 s res: 480.629 MB peak: 962.625 MB
v2 ---> time: 0.785457 s res: 151.215 MB peak: 152.301 MB


====== 4 - Training unindexed PatternModel from file: threshold 2, up to 8-grams, no skipgrams======
v1 ---> time: 375.626 s res: 1174.49 MB peak: 1656.11 MB
v2 ---> time: 313.736 s res: 965.391 MB peak: 1117.48 MB

====== 5 - Training unindexed PatternModel from preloaded corpus: threshold 2, up to 8-grams, no skipgrams======
v1 ---> time: 381.641 s res: 1174.49 MB peak: 1656.1 MB
v2 ---> time: 269.111 s res: 964.723 MB peak: 1116.75 MB

====== 6 - Training unindexed PatternModel from file: threshold 2, up to 8-grams, with skipgrams (exhaustive)======
v1 ---> time: 14157.6 s res: 48953.1 MB peak: 49434.8 MB
v2 ---> time: 2027.98 s res: 8679.61 MB peak: 8831.7 MB

====== 7 - Training unindexed PatternModel from preloaded corpus: threshold 2, up to 8-grams, with skipgrams (exhaustive)======
v1 ---> time: 14170.5 s res: 48953.1 MB peak: 49434.8 MB
v2 ---> time: 2005.42 s res: 8676.69 MB peak: 8828.78 MB

====== 8 - Training unindexed PatternModel from file: threshold 1, up to 8-grams, no skipgrams======
v1 ---> time: 453.207 s res: 14282.8 MB peak: 15436.4 MB
v2 ---> time: 567.805 s res: 11901.1 MB peak: 12812.5 MB

====== 9 - Training indexed PatternModel from preloaded corpus: threshold 2, up to 8-grams, no skipgrams======
v1 ---> time: 457.89 s res: 2986.92 MB peak: 3468.54 MB
v2 ---> time: 343.331 s res: 2788.84 MB peak: 2940.87 MB

====== 10 - Training indexed PatternModel from preloaded corpus: threshold 2, up to 8-grams, with skipgrams======
v1 ---> time: 1336.39 s res: 6159.37 MB peak: 6640.98 MB
v2 ---> time: 1733.88 s res: 11650.9 MB peak: 11802.9 MB

====== 11 - Training unindexed PatternPointerModel from preloaded corpus: threshold 2, up to 8-grams, no skipgrams======
v2 ---> time: 232.662 s res: 757.82 MB peak: 909.914 MB

====== 12 - Training unindexed PatternPointerModel from preloaded corpus: threshold 2, up to 8-grams, with skipgrams (exhaustive)======
v2 ---> time: 1609.68 s res: 6000.96 MB peak: 6153.05 MB

====== 13 - Training indexed PatternPointerModel from preloaded corpus: threshold 2, up to 8-grams, no skipgrams======
v2 ---> time: 282.332 s res: 2558.21 MB peak: 2710.36 MB

====== 14 - Training unindexed PatternModel<HashOrderedPatternMap> from preloaded corpus: threshold 2, up to 8-grams, no skipgrams======
v2 ---> time: 1016.27 s res: 1241.98 MB peak: 1394.13 MB