Skip to content

Instantly share code, notes, and snippets.

@carlhoerberg
Created April 17, 2012 15:00
Show Gist options
  • Save carlhoerberg/2406557 to your computer and use it in GitHub Desktop.
Save carlhoerberg/2406557 to your computer and use it in GitHub Desktop.
Concurrent HTTP requests with EventMachine
require 'eventmachine'
require 'em-http-request'
# Reference:
# https://github.com/igrigorik/em-http-request/wiki/Parallel-Requests
# http://rdoc.info/github/eventmachine/eventmachine/master/EventMachine/Iterator
urls = ['http://www.google.com', 'http://www.cloudamqp.com']
# always have 100 open request at any given time
concurrency = 100
EM.run do
EM::Iterator.new(urls, concurrency).each(
proc { |url, iter|
http = EventMachine::HttpRequest.new(url).get
http.callback do |response|
p response.response
iter.next
end
http.errback do
p "Failed: #{url}"
iter.next
end
},
proc {
p 'All done!'
EM.stop
})
end
@l1x
Copy link

l1x commented Mar 30, 2013

Is there a way to have a certain amount of connections open at a time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment