Skip to content

Instantly share code, notes, and snippets.

@RickyCook
Last active August 29, 2015 14:08
Show Gist options
  • Save RickyCook/dd0cbaae8de1fee433e4 to your computer and use it in GitHub Desktop.
Save RickyCook/dd0cbaae8de1fee433e4 to your computer and use it in GitHub Desktop.
Test the "latency" of different ways to trigger multiprocessing start
"""
Test the "latency" of different ways to trigger multiprocessing start.
This will essentially create a multiprocessing `Process` object and measure the
time from when we tell that process to start processing (not necessarily
`Process.start`) until the time that the process picks up the request to start.
This is done in several ways:
- Without a blocker: the time is recorded, then the `Process.start` method
is used to start the fork, which simply records its as the first, and only
task that it performs (measuring the time between start and record)
- With a Lock object: a Lock is created, acquired, and sent to the `Process`
object as an arg. The `Process` is started and waits to acquire the
`Lock`. In the main process, the time is recorded the lock is released. In
the fork, the time is then recorded (measuring the time between unlock and
record)
My first run:
$ uname -a
Linux blaze 3.16-2-amd64 #1 SMP Debian 3.16.3-2 (2014-09-20) x86_64 GNU/Linux
$ python3 --version
Python 3.4.2rc1+
$ python3 multiprocessing_latency.py
Total without blocker: 16.47618007659912
Total with lock: 13.602282047271729
$ python2 --version
Python 2.7.8
$ python2 multiprocessing_latency.py
Total without blocker: 13.8391010761
Total with lock: 11.4278309345
"""
from __future__ import print_function
import ctypes
import multiprocessing
import time
ITERATIONS = 10000
def no_blocker_job(vshared):
thetime = time.time()
vshared.value = thetime
def lock_job(vshared, lock):
lock.acquire()
thetime = time.time()
vshared.value = thetime
def test_no_blocker():
vshared = multiprocessing.Value(ctypes.c_double)
proc = multiprocessing.Process(target=no_blocker_job, args=(vshared,))
starttime = time.time()
proc.start()
proc.join()
return vshared.value - starttime
def test_lock():
vshared = multiprocessing.Value(ctypes.c_double)
lock = multiprocessing.Lock()
lock.acquire()
proc = multiprocessing.Process(target=lock_job, args=(vshared, lock))
proc.start()
starttime = time.time()
lock.release()
proc.join()
return vshared.value - starttime
def main():
total_no_blocker = total_lock = 0.0
out_step = ITERATIONS/100
for i in range(ITERATIONS):
if i % out_step == 0:
print("%d%% done" % (i / out_step))
total_no_blocker += test_no_blocker()
total_lock += test_lock()
print("Total without blocker: %s" % total_no_blocker)
print("Total with lock: %s" % total_lock)
if __name__ == '__main__':
main()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment