Skip to content

Instantly share code, notes, and snippets.

@sdhutchins
Forked from hoffrocket/parhttp.py
Created March 27, 2017 23:12
Show Gist options
  • Save sdhutchins/91f56ccf93c4015071035ff23639717b to your computer and use it in GitHub Desktop.
Save sdhutchins/91f56ccf93c4015071035ff23639717b to your computer and use it in GitHub Desktop.
Python parallel http requests using multiprocessing
#!/usr/bin/env python
from multiprocessing import Process, Pool
import time
import urllib2
def millis():
return int(round(time.time() * 1000))
def http_get(url):
start_time = millis()
result = {"url": url, "data": urllib2.urlopen(url, timeout=5).read()[:100]}
print url + " took " + str(millis() - start_time) + " ms"
return result
urls = ['http://www.google.com/', 'https://foursquare.com/', 'http://www.yahoo.com/', 'http://www.bing.com/', "https://www.yelp.com/"]
pool = Pool(processes=5)
start_time = millis()
results = pool.map(http_get, urls)
print "\nTotal took " + str(millis() - start_time) + " ms\n"
for result in results:
print result
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment