Skip to content

Instantly share code, notes, and snippets.

@dmitryhd
Last active December 2, 2015 11:05
Show Gist options
  • Select an option

  • Save dmitryhd/39e68162219b60e82a9b to your computer and use it in GitHub Desktop.

Select an option

Save dmitryhd/39e68162219b60e82a9b to your computer and use it in GitHub Desktop.
Multiprocessing in Python
from multiprocessing import Process
from time import sleep
import pandas as pd
series = pd.Series(range(100))
data = series[:3]
def process_chunk(chunk_id, chunk_size):
local_data = data[chunk_id * chunk_size: (chunk_id + 1) * chunk_size]
for x in local_data:
print(chunk_id, x)
process_pool = []
num_proc = 2
chunk_size = len(data)/num_proc + 1
for i in range(num_proc):
proc = Process(target=process_chunk, args=(i, chunk_size))
process_pool.append(proc)
for proc in process_pool:
proc.start()
for proc in process_pool:
proc.join()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment