Skip to content

Instantly share code, notes, and snippets.

@vishnugopal

vishnugopal/Set

Created February 23, 2011 10:21
Show Gist options
  • Save vishnugopal/840246 to your computer and use it in GitHub Desktop.
Save vishnugopal/840246 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Set
wc -l 14.csv = 7631928
wc -l base.csv = 1000000
Bitmap:
ruby -rubygems fast_filter_shell.rb add --engine bitmap 14.csv 556.92s user 152.28s system 63% cpu 18:41.26 total
ruby -rubygems fast_filter_shell.rb filter --engine bitmap base.csv 73.79s user 20.08s system 67% cpu 2:18.51 total
Space: 570MB (constant)
Bloom:
ruby -rubygems fast_filter_shell.rb add --engine bloom 14.csv 4466.57s user 1621.11s system 58% cpu 2:53:27.80 total
ruby -rubygems fast_filter_shell.rb filter --engine bloom base.csv 356.81s user 128.92s system 61% cpu 13:12.04 total
Space: 350MB (constant, eager loaded)
Set:
ruby -rubygems fast_filter_shell.rb add --engine set 14.csv 491.68s user 153.02s system 61% cpu 17:33.70 total
ruby -rubygems fast_filter_shell.rb filter --engine set base.csv 68.48s user 20.08s system 63% cpu 2:18.44 total
Space: 410 MB (scales with size)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment