Created
January 21, 2013 21:07
-
-
Save Velrok/4589404 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from multiprocessing import Pool, cpu_count | |
def multi_process_map(fn, args): | |
'''Applyes function fn to each element in args running in a seperate process. | |
This will block until all function calls are finished. | |
This uses a process pool, which size equals the number of available cpu's. | |
fn: a funtion that takes one argument | |
args: a list of arguments to that function | |
returns: a listof results''' | |
pool = Pool(cpu_count()) | |
return pool.map(fn, args) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This uses multi process because of this:
CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.