Multiprocessing with Selenium and Python for website crawling

selenium_multiprocess.py

from multiprocessing import Pool, cpu_count

def run_parallel_selenium_processes(datalist, selenium_func):

    pool = Pool()

    # max number of parallel process
    ITERATION_COUNT = cpu_count()-1

    count_per_iteration = len(datalist) / float(ITERATION_COUNT)

    for i in range(0, ITERATION_COUNT):
        list_start = int(count_per_iteration * i)
        list_end = int(count_per_iteration * (i+1))
        pool.apply_async(selenium_func, [datalist[list_start:list_end]])

Thank you for the example.

I have a question: how do we get the result returned? When I apply get() method on pool.apply_async, the web driver starts in sequence, not simultaneously.

Could you help me with this problem, please?

I solved this by adding this line of code in the for loop:
results = []
results.extend(pool.apply_async(selenium_func, [datalist[list_start:list_end]]))

and by this before return
results = [results[i].get() for i in range(len(results))]

Thank you