1000 concurrent downloads: gevent vs multiprocessing cc1.4xlarge

gistfile1.sh

ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Multiprocessing:
Took 2.014980793 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Multiprocessing:
Took 6.96465492249 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Multiprocessing:
Took 5.79786896706 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Multiprocessing:
Took 1.47629785538 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Multiprocessing:
Took 1.73595380783 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Multiprocessing:
Took 2.13242602348 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Multiprocessing:
Took 1.65409302711 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Multiprocessing:
Took 1.82931685448 seconds...

ubuntu@ip-10-17-167-43:~$ vi downloading_complete.py 
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Gevent:
Took 5.2130188942 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Gevent:
Took 2.21397590637 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Gevent:
Took 2.58203887939 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Gevent:
Took 4.89094281197 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Gevent:
Took 2.24247407913 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Gevent:
Took 2.5157289505 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Gevent:
Took 2.59691905975 seconds...
ubuntu@ip-10-17-167-43:~$ ./downloading_complete.py 
With Gevent:
Took 2.39316606522 seconds...

So, what you're saying is that multiple processes are better! I guess, in reality, gevent is better on average, but the average is killed by the 6 second craziness for multiprocessing. Other statistics would be more informative, but I've lost interest.

Yup. Pretty much! Next step is to try multiple gevented processes.

…

On Mar 13, 2012, at 6:11 AM, Andrew Gwozdziewycz ***@***.*** wrote: So, what you're saying is that multiple processes are better! --- Reply to this email directly or view it on GitHub: https://gist.github.com/2027568

Well, multiple cores anyway. Multiple processes on limited cores did quite badly. Check out gevent vs. multiprocess on one core here: https://gist.github.com/2027506