words_filter.rb

# encoding: utf-8
text = %{Her mission may not have been accomplished though, as Styles took the high road when asked about Swift and her performance on a British radio show this week.
"She's a great performer and she always performs great," he praised. "She's always good on the stage. She's been doing it a long time. She knows what she's doing on stage. It was just another good Taylor Swift performance."

Swift and Harry Styles (Alequin/Bosch/INFDaily.com)
He also wasn’t concerned about running into her at the Brit Awards, where they were both scheduled to perform on Wednesday night.
"I'm not worried about seeing her at all," he recently told Grazia magazine. "She's a sweet girl, you know, I don't have a bad word to say about her."
Despite Styles' good-guy vibe, it seems that Swift may have been trying to send more messages the One Direction singer’s way when she took the stage at the Brit Awards where Styles happened to be sitting in the audience. The blonde pranced around in a white poofy dress that looked a whole lot like a wedding gown before ripping it off to reveal a skimpy black-hot-pants outfit paired with laceup boots. Translation: I didn’t want to marry you anyway, Harry?
We may never know for sure, but given Swift’s history, chances are the move was indeed another dig at an ex. In the past, most of her swipes, however, come via her song lyrics and media interviews, where she hasn’t been afraid to shame the dudes who left her behind.}

ESCAPE_1 = %w{have has is are had to a an by on off and in of from for were as be with was at been no not also neither do doing}
ESCAPE_2 = %w{his her i we he she my it the s t m you your us they}
ESCAPE_3 = %w{how however where when who that but let make may this what}
ESCAPE_4 = %w{before behind after since about via}
ESCAPE_5 = %w{much more often always just both never most}
ESCAPE_6 = %w{out inside down up left right into}
NON = %w{hasn don wasn didn}
PERSON = %w{ms mr}
NUMBER = %w{one two three four five six seven eight nine ten third}
DATE = %w{monday tuesday wednesday}

ESCAPE = ESCAPE_1+ESCAPE_2+ESCAPE_3+ESCAPE_4+ESCAPE_5+ESCAPE_6+NUMBER+DATE+NON
reg = ESCAPE.join("|")
#full name / geo name
f_name = text.scan(/[[:upper:]]\w+\s[[:upper:]]\w+/)
text = text.gsub(/[[:upper:]]\w+\s[[:upper:]]\w+/,'')
p f_name
words = text.scan(/[a-zA-Z0-9_-]+/)
#p words
text = words.map{|x| "<#{x}>"}.join
#p text
# remove uip-words and digit
text = text.gsub(/<(#{reg}|\d+)>/i,'')
#p text
 
# token parse
# match people name
mr = text.scan(/<Mr><[a-zA-Z0-9_-]+>/)
ms = text.scan(/<Ms><[a-zA-Z0-9_-]+>/)
people = mr+ms
m_person = people.uniq.map{ |x| [x,people.grep(x).length] }.sort!{|a,b| b[1] <=> a[1]}
text = text.gsub(/<Mr><[a-zA-Z0-9_-]+>/,'').gsub(/<Ms><[a-zA-Z0-9_-]+>/,'')

upcase_words  = text.scan(/<[[:upper:]]\w+>/)
text = text.gsub(/<[[:upper:]]\w+>/,'')

text = text.gsub(/(<|>)/,' ')
#p text
words = text.scan(/[a-zA-Z0-9_-]+/)
#p words

words = words.uniq.map{ |x| [x,words.grep(x).length] }

words.sort!{|a,b| b[1] <=> a[1]}.each{|s| p s}
p words.length

text.scan(/<[A-Z]\w+>/)