Python threading comparison with listdir. Refactored to pull out the threading stuff from the measurements. This work also tries to show when one blocks between put/get it can really slow things down, and it gets worse across multiple CPUs. Run on a ThinkPad T520

1cpu.txt

$ numactl -C 0 python b.py

Benchmark Report
================

BenchQueueStuff
---------------

                        name | rank | runs |    mean |       sd | timesBaseline
-----------------------------|------|------|---------|----------|--------------
               one at a time |    1 |   50 | 0.05127 | 0.003448 |           1.0
                   big batch |    2 |   50 | 0.05361 |  0.01131 | 1.04551505778
    producer consumer thread |    3 |   50 | 0.05523 | 0.005709 | 1.07720172854
producer consumer thread 100 |    4 |   50 | 0.05648 | 0.006446 | 1.10159528409
 producer consumer thread 10 |    5 |   50 |  0.1252 |  0.01642 | 2.44255953358
                     listdir |    6 |   50 |  0.1631 | 0.009314 | 3.18123328994
  producer consumer thread 1 |    7 |   50 |   1.116 |   0.1188 | 21.7596150278

Each of the above 350 runs were run in random, non-consecutive order by
`benchmark` v0.1.5 (http://jspi.es/benchmark) with Python 2.7.3
Linux-3.10.10-100.fc18.x86_64-x86_64 on 2013-09-19 19:55:17.

b.py

#!/usr/bin/env python

import benchmark
import Queue
import threading
import os
from time import sleep


class BenchQueueStuff(benchmark.Benchmark):
    def setUp(self):
        self.size = 10000

    def eachSetUp(self):
        self.queue = Queue.Queue()
        self.tqueue = Queue.Queue()
        def consume():
            for _ in xrange(self.size):
                x = self.tqueue.get()
                if x == -1:
                    return
        self.thr = threading.Thread(target=consume)
        self.thr.daemon = True
        self.thr.start()
        # Yield to let the first thread "get" into the queue.
        sleep(0.00001)

    def eachTearDown(self):
        # For tests that don't use tqueue
        self.tqueue.put(-1)
        self.thr.join()

    def test_big_batch(self):
        for i in xrange(self.size):
            self.queue.put(i)
        for _ in xrange(self.size):
            self.queue.get()

    def test_one_at_a_time(self):
        for i in xrange(self.size):
            self.queue.put(i)
            self.queue.get()

    def test_listdir(self):
        cwd = os.getcwd()
        for i in xrange(self.size):
            os.listdir(cwd)

    def test_producer_consumer_thread(self):
        for i in xrange(self.size):
            self.tqueue.put(i)
        self.thr.join()

    def test_producer_consumer_thread_1(self):
        for i in xrange(self.size):
            self.tqueue.put(i)
            sleep(0.00001)
        self.thr.join()

    def test_producer_consumer_thread_10(self):
        for i in xrange(self.size):
            self.tqueue.put(i)
            if (i % 10) == 0:
                sleep(0.00001)
        self.thr.join()

    def test_producer_consumer_thread_100(self):
        for i in xrange(self.size):
            self.tqueue.put(i)
            if (i % 100) == 0:
                sleep(0.00001)
        self.thr.join()


if __name__ == '__main__':
    benchmark.main(each=50, numberFormat="%.4g")

lscpu.txt

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Stepping:              7
CPU MHz:               800.000
BogoMIPS:              5382.82
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              4096K
NUMA node0 CPU(s):     0-3

Ncpu.txt

Benchmark Report
================

BenchQueueStuff
---------------

                        name | rank | runs |    mean |       sd | timesBaseline
-----------------------------|------|------|---------|----------|--------------
                   big batch |    1 |   50 | 0.05364 | 0.005286 |           1.0
               one at a time |    2 |   50 | 0.05401 | 0.004689 | 1.00692688833
    producer consumer thread |    3 |   50 |  0.1443 |  0.06164 | 2.69027391099
                     listdir |    4 |   50 |  0.1642 | 0.007942 | 3.06041988899
producer consumer thread 100 |    5 |   50 |  0.2387 |  0.08739 | 4.44988755196
 producer consumer thread 10 |    6 |   50 |  0.4317 |   0.1291 | 8.04803513712
  producer consumer thread 1 |    7 |   50 |   1.149 |  0.06077 | 21.4134447858

Each of the above 350 runs were run in random, non-consecutive order by
`benchmark` v0.1.5 (http://jspi.es/benchmark) with Python 2.7.3
Linux-3.10.10-100.fc18.x86_64-x86_64 on 2013-09-19 19:57:19.