Hello all. We must choose an extraction library for our new Goliath system to provide default values when no rules have been set. The choice has been narrowed down to Boilerpipe and Goose. They both have sub-par documenation (Boilerpipe, Goose), so I've dug around in the code to find the exact process by which they pull out data. Here I will compare them so we can choose one.
#Boilerpipe
This bad mamba jamba was developed by a Ph.D-having guy who, along with some other folks, wrote a big fat academic paper around the algorithm it uses, which you can find in our Dropbox if you really want to read it. Basically, they use link density, text density, and number of words on a block-by-block basis to distinguish boilerplate blocks from content blocks. A block is simply a contiguous piece of text terminated by the start
