Quick and dirty way to find nuplets word frequency in a series of texts

word-frequency-nuplet-1.rb

# coding utf-8
=begin

Working on y blog internal linking, I started to think about a better than just tags way to find similarities between texts. 

I wanted to find 2 and 3 words similarities betwen my posts, so I wrote that quick and dirty script to do it. I'm then comparing the token appearing 2 to 5 times with the tags used in the article to build my "related posts" link.

Results are not great so far, because I'm working with only a 1200 articles with very unique content. Next step is to include a list of synonymouses to get better results.

=end

require 'tactful_tokenizer'

class String
  TAG_KEY = ATTRIBUTE_KEY = /[\w:_-]+/
  ATTRIBUTE_VALUE = /(?:[A-Za-z0-9]+|(?:'[^']*?'|"[^"]*?"))/
  ATTRIBUTE = /(?:#{ATTRIBUTE_KEY}(?:\s*=\s*#{ATTRIBUTE_VALUE})?)/
  ATTRIBUTES = /(?:#{ATTRIBUTE}(?:\s+#{ATTRIBUTE})*)/
  TAG = %r{<[!/?\[]?(?:#{TAG_KEY}|--)(?:\s+#{ATTRIBUTES})?\s*(?:[!/?\]]+|--)?>}

  def strip_html
    self.gsub!(TAG, '').gsub(/\s+/, ' ').strip
  end
end

def word_frequency(text, nuplet)
  m = TactfulTokenizer::Model.new
  glossary = Hash.new
  
  clean_text = m.tokenize_text text.strip_html
  
  clean_text.each do |sentence|
    counter = 0
    words = sentence.split
    
    while words.size >= nuplet
      token = words.take(nuplet).join(' ')
      glossary[token] = glossary.has_key?(token) ? glossary[token] + 1 : 1
      words.shift
    end
  end
end

glossary = Hash.new

texts.each do |text|
  # find triplets frequency
  nuplets = word_frequency(text.body_and_extended.strip_html, 3)
  
  nuplets.each do |token, value|
    glossary[token] = glossary.has_key?(token) ? glossary[token] + value : value
  end
end

glossary.sort do |a,b| 
  a[1] <=> b[1]}.each do |elem|
    puts "\"#{elem[0]}\" has #{elem[1]} occurrences"
  end
end