Skip to content

Instantly share code, notes, and snippets.

@Velrok
Created January 21, 2013 21:07
Show Gist options
  • Save Velrok/4589404 to your computer and use it in GitHub Desktop.
Save Velrok/4589404 to your computer and use it in GitHub Desktop.
from multiprocessing import Pool, cpu_count
def multi_process_map(fn, args):
'''Applyes function fn to each element in args running in a seperate process.
This will block until all function calls are finished.
This uses a process pool, which size equals the number of available cpu's.
fn: a funtion that takes one argument
args: a list of arguments to that function
returns: a listof results'''
pool = Pool(cpu_count())
return pool.map(fn, args)
@Velrok
Copy link
Author

Velrok commented Jan 21, 2013

This uses multi process because of this:

CPython implementation detail: In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment