Skip to content

Instantly share code, notes, and snippets.

@yannick
Created May 11, 2011 19:05
Show Gist options
  • Select an option

  • Save yannick/967080 to your computer and use it in GitHub Desktop.

Select an option

Save yannick/967080 to your computer and use it in GitHub Desktop.
#!/Volumes/data/home/yannick/.rvm/rubies/ruby-1.9.2-preview3/bin/ruby
require 'open-uri'
require 'nokogiri'
save_dir = "./html/"
url = 'http://www.admin.ch/ch/d/sr/101/index.html'
toplink = url.gsub(/index.html$/,'')
doc = Nokogiri::HTML(open(url))
toc = doc.css('#spalteContentPlus')
layer1_links = toc.css('a').collect{|lnk| lnk["href"]}.select{|lnk| lnk.to_s.start_with? "./"}
layer1_links.each do |lnk|
lnk.gsub!(/^.\//,'')
subsite = toplink + lnk
puts "fetching #{subsite}"
sdoc = Nokogiri::HTML(open(subsite))
text = sdoc.css('#spalteContentPlus')
filename = save_dir + lnk
puts "writing to #{filename}"
File.open( filename, 'w'){ |file|
file.write text.to_html
}
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment