clean-html-detect-domains.rb

require 'nokogiri'
require 'uri'
require 'resolv'

raw = <<-EOS
<p><span class="KonaFilter image-container display-table float_right"><span><span data-post-image="" class="image on-image"><img src="http://static3.businessinsider.com/image/56b3c07b6e97c621048b4fbe-975-731/4278432941_5cb085182e_b.jpg" alt="linkedin chocolates" data-mce-source="Nan Palmero/Flickr" data-link="https://www.flickr.com/photos/nanpalmero/4278432941/in/photolist-7w564c-9fLRVv-7yuDWB-n5ztS-d59ToJ-5BwY81-82Buin-82EzHN-cNL4r7-82EAmu-cNL2KS-5R4hW5-9MKoLL-adWYuV-spAUPG-ayeJFw-yb7CFx-dyyYT5-y1k6uo-8W2L2-aFbk4G-dTZsva-9v76Un-sG1A7b-dwtTSy-9Y9nGA-eqJboT-zX9Xu-7xatis-AV3XRq-axKupX-gjuvyS-8Q7Mti-ahEZH8-82EA7f-s98dqU-8QaSYo-arq76A-aEVD3h-7RD8PQ-dQuDXo-ayc3yD-e2G9qu-dQp4Xr-6tLekM-n2HCV-beKTxv-rPWtoe-dq4wxW-cXgNKw"><span class="source"><span><a href="https://www.flickr.com/photos/nanpalmero/4278432941/in/photolist-7w564c-9fLRVv-7yuDWB-n5ztS-d59ToJ-5BwY81-82Buin-82EzHN-cNL4r7-82EAmu-cNL2KS-5R4hW5-9MKoLL-adWYuV-spAUPG-ayeJFw-yb7CFx-dyyYT5-y1k6uo-8W2L2-aFbk4G-dTZsva-9v76Un-sG1A7b-dwtTSy-9Y9nGA-eqJboT-zX9Xu-7xatis-AV3XRq-axKupX-gjuvyS-8Q7Mti-ahEZH8-82EA7f-s98dqU-8QaSYo-arq76A-aEVD3h-7RD8PQ-dQuDXo-ayc3yD-e2G9qu-dQp4Xr-6tLekM-n2HCV-beKTxv-rPWtoe-dq4wxW-cXgNKw">Nan Palmero/Flickr</a></span></span></span></span></span></p><p>LinkedIn's stock had&nbsp;its worst day ever.</p>
<p>Shares cratered by as much as 44% in trading on Friday after the company reported a quarterly loss on Thursday, with guidance that was weaker than expected.</p>
<p>It was the worst decline ever for the stock and has erased about $10 billion of the company's value.</p>
<p>The social network for professionals said that it expected first-quarter adjusted earnings per share (EPS) of $0.55, but analysts had estimated $0.75, according to Bloomberg.</p>
<p>It projected full-year revenues at $3.6 billion to $3.65 billion, versus $3.9 billion expected.</p>
<p>LinkedIn reported revenue of $861.9 million for the fourth quarter, and its earnings loss was $0.06 a share, or $8.4 million.</p>
<p>It recorded year-over-year growth in revenues in its major business segments, including a 22% rise in premium subscriptions.</p>
<p>The company said in its earnings <a href="http://finance.yahoo.com/news/linkedin-announces-fourth-quarter-full-210500357.html">release</a> that the number of cumulative members grew 19% to 414 million, with mobile visits now making up 57% of all traffic. In December, it launched an updated version of its app.</p>
<p>LinkedIn also announced that it was killing Lead Accelerator, a business-to-business ad network it got with the 2014 acquisition of Bizo.</p>
<p>Pacific Crest's Evan Wilson ([email protected]) and Tyler Parker did not like this decision.</p>
<p><span>They said in a note on Thursday:</span></p>
<p style="padding-left: 30px;">We would understand if the ad network had grave privacy issues or if there was no demand, but other examples show us this is not the case.&nbsp;Now Sales Navigator really is LinkedIn's only big near-term opportunity to materially increase the monetization of its data set. Ugh.</p>
<p>Over the past year, LinkedIn (www.linkedin.com) shares have fallen by about 52%.</p>
<p>"We did not learn our lesson," the Pacific Crest analysts wrote. "LinkedIn has rarely given investors a reason to own the stock in 1H and it has happened again. It will have to spend the balance of the year restoring confidence in a much tougher environment."</p>
<p>Here's the drop in trading on Friday to the lowest level in three years:</p>
<p><span class="KonaFilter image-container display-table click-to-enlarge"><span><span data-post-image="" class="image"><a title_source="Google" title_caption="" class="zoomin" href="http://static1.businessinsider.com/image/56b4b9256e97c686008b520d-960/screen%20shot%202016-02-05%20at%2010.00.05%20am.png"><span class="sprites click-to-enlarge-icon" title="Click to enlarge"></span><img src="http://static1.businessinsider.com/image/56b4b9256e97c686008b520d-1045-703/screen%20shot%202016-02-05%20at%2010.00.05%20am.png" alt="Screen Shot 2016 02 05 at 10.00.05 AM" data-mce-source="Google"></a><span class="source"><span>Google</span></span></span></span></span></p><p></p>
EOS

text = Nokogiri::HTML(raw).text

urls = URI.extract(text).uniq
urls.each do |url|
  text = text.gsub(url, '')
end

tokens = text.split(/\p{Zs}+/)
domains = tokens.map { |token| token.match(/([\p{L}-]+\.)+\p{L}[\p{L}-]+/) }.compact.map { |d| d.to_s }.uniq
resolved = Hash[domains.map { |domain| [domain, (Resolv.getaddress domain rescue nil)] }]
domains = resolved.reject { |k,v| v.nil? }.keys

puts "DOMAINS:"
puts domains.join("\n")

tokens = text.split(/[^\S\n]+/)
text = tokens.join(' ')
text = text.gsub(/^$/, '').strip
text = text.gsub(/^$[\s]*/, '')

puts "TEXT:"
puts text