pandas DataFrame apply multiprocessing

apply_df_by_multiprocessing.py

import multiprocessing
import pandas as pd
import numpy as np

def _apply_df(args):
    df, func, kwargs = args
    return df.apply(func, **kwargs)

def apply_by_multiprocessing(df, func, **kwargs):
    workers = kwargs.pop('workers')
    pool = multiprocessing.Pool(processes=workers)
    result = pool.map(_apply_df, [(d, func, kwargs)
            for d in np.array_split(df, workers)])
    pool.close()
    return pd.concat(list(result))
    
def square(x):
    return x**x
    
if __name__ == '__main__':
    df = pd.DataFrame({'a':range(10), 'b':range(10)})
    apply_by_multiprocessing(df, square, axis=1, workers=4)  
    ## run by 4 processors
  

Another note worth adding:
I wanted to pass an argument to the function I wanted to apply. I found the best way to do that with the suggested implementation is using functools.partial, which creates pickleable objects, so the application would look like:

apply_by_multiprocessing(df, functools.parial(my_func, some_arg), axis=1, workers=4)  

I'm getting this error
PicklingError: Can't pickle <function _apply_df at 0x7f2c2e876158>: attribute lookup _apply_df on main failed

Thanks for the Code, have a good idea from this. But I need to send two data frames and one pandas.core.groupby.generic.DataFrameGroupBy object to my_function(). How I can achieve this through multiprocessing. if anyone can share thought / expertise, that will be great help.

Thanks for the Code, have a good idea from this. But I need to send two data frames and one pandas.core.groupby.generic.DataFrameGroupBy object to my_function(). How I can achieve this through multiprocessing. if anyone can share thought / expertise, that will be great help.

Check this https://github.com/zahrashuaib/parallel-computing. The dataframe sent to the function for the multiprocess.

I wrote a package to use apply methods on Series, DataFrames and GroupByDataFrames on multiple cores. It makes it very easy to do multiprocessing in Pandas.

You can check the documentation at https://github.com/akhtarshahnawaz/multiprocesspandas

You can also install the package directly using pip

pip install multiprocesspandas

Then doing multiprocessing is as simple as importing the package as

from multiprocesspandas import applyparallel

and then using applyparallel instead of apply like

def func(x):
    import pandas as pd
    return pd.Series([x['C'].mean()])

df.groupby(["A","B"]).apply_parallel(func, num_processes=30)