Strange mpi4py error.

README.md

The script works.py runs fine and give reasonable results but the script notworking.py throws a cPickle.UnpicklingError and then stalls. The only difference is the line:

nwalkers = 100  # Works.
nwalkers = 150  # Doesn't work.

notworking.py

import sys
import numpy as np
import emcee
from emcee.utils import MPIPool


def lnprob(x):
    return -0.5 * np.sum(x ** 2)

ndim = 50
nwalkers = 150
p0 = [np.random.rand(ndim) for i in xrange(nwalkers)]
pool = MPIPool()

if not pool.is_master():
    # Wait for instructions from the master process.
    pool.wait()
    sys.exit(0)

sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, pool=pool)
pos, prob, state = sampler.run_mcmc(p0, 1)

works.py

import sys
import numpy as np
import emcee
from emcee.utils import MPIPool


def lnprob(x):
    return -0.5 * np.sum(x ** 2)

ndim = 50
nwalkers = 100
p0 = [np.random.rand(ndim) for i in xrange(nwalkers)]
pool = MPIPool()

if not pool.is_master():
    # Wait for instructions from the master process.
    pool.wait()
    sys.exit(0)

sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, pool=pool)
pos, prob, state = sampler.run_mcmc(p0, 1)

I'm experiencing similar cPickle.UnpicklingError with simpler snippets of code:

import emcee
import numpy as np

pool = emcee.utils.MPIPool()

def f(x): return np.empty(2500)

X = np.empty([100,50000])

Y = np.array(pool.map(f, [x for x in X]))

pool.close()

If I reduce the size of the array returned by f from 2500 to 100 for instance, the code just works. This issue might be related to MPI buffer size limits.