Skip to content

Instantly share code, notes, and snippets.

@namnv609
Created September 10, 2018 02:35
Show Gist options
  • Save namnv609/8108ccb4b49f265f8eba16b31f1a8caa to your computer and use it in GitHub Desktop.
Save namnv609/8108ccb4b49f265f8eba16b31f1a8caa to your computer and use it in GitHub Desktop.
Simple crawler for DevVui fb fanpage
require "unirest"
require "nokogiri"
require "pry"
fb_url = "https://m.facebook.com/devvui/"
ua = "Mozilla/5.0 (Windows Phone 10.0; Android 6.0.1; WebView/3.0; NOKIA; Lumia 730 Dual SIM) AppleWebKit/537.36 (KHTML, like Gecko) coc_coc_browser/55.100.7 Chrome/48.0.2564.82 Mobile Safari/537.36 Edge/14.14361"
html = Unirest.get fb_url, headers: {"User-Agent": ua}
doc = Nokogiri::HTML html.body
# doc.xpath("//a/img[@class='img']").each do |c|
doc.xpath("//div[contains(@class, 'fr')]/a[contains(@class, 'fs')]").each do |c|
puts c['href']
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment