Skip to content

Instantly share code, notes, and snippets.

@gdisneyleugers
Created November 18, 2013 02:17
Show Gist options
  • Save gdisneyleugers/7521369 to your computer and use it in GitHub Desktop.
Save gdisneyleugers/7521369 to your computer and use it in GitHub Desktop.
require 'anemone'
require 'redis'
require 'uuidtools'
redis = Redis.new(:host => "localhost", :port => 6379)
puts "URL to crawl: "
url = gets
Anemone.crawl(url) do |anemone|
anemone.on_every_page do |page| key=UUIDTools::UUID.random_create
puts key
puts page.url
redis.set(key, page.url)
redis.get(key)
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment