Skip to content

Instantly share code, notes, and snippets.

@masahitojp
Created November 30, 2011 07:54
Show Gist options
  • Save masahitojp/1408382 to your computer and use it in GitHub Desktop.
Save masahitojp/1408382 to your computer and use it in GitHub Desktop.
multiprocessing.Poolを使ってHTTPリクエストを多重に投げる。(スクレイピングとかには絶対使わないこと)
import json
import urllib2
import httplib2
import multiprocessing
url='url'
data = json.dumps({'id':'username', 'password':'password'})
def start_process():
print 'Starting', multiprocessing.current_process().name
def worker(num):
print 'Worker:', num
http=httplib2.Http()
header_data={'Content-Type': 'application/json'}
response, content = http.request(url, 'POST',
headers=header_data, body=data)
print(response)
print(content)
return
if __name__ == '__main__':
pool_size = multiprocessing.cpu_count()
print pool_size
pool = multiprocessing.Pool(processes=pool_size)
pool_outputs = pool.map_async(worker, range(100))
pool.close() # no more tasks
pool.join() # wrap up current tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment