Skip to content

Instantly share code, notes, and snippets.

@gdisneyleugers
Created November 18, 2013 00:27
Show Gist options
  • Save gdisneyleugers/7520320 to your computer and use it in GitHub Desktop.
Save gdisneyleugers/7520320 to your computer and use it in GitHub Desktop.
require 'anemone'
require 'redis'
key=rand(0..17179869184)
puts key
redis = Redis.new(:host => "0.0.0.0", :port => 6379)
puts "URL to crawl: "
url = gets
Anemone.crawl(url) do |anemone|
anemone.on_every_page do |page|
puts page.url
redis.set(key, page.url)
redis.get(key)
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment