Skip to content

Instantly share code, notes, and snippets.

@dokipen

dokipen/pysparkout.txt

Created June 10, 2015 15:15
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save dokipen/84c4e4a89fddf702fdf1 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save dokipen/84c4e4a89fddf702fdf1 to your computer and use it in GitHub Desktop.
Download ZIP
pyspark PYTHONPATH issue
Raw
pysparkout.txt
$ PYTHONPATH="$PYTHONPATH:DOESNT_GET_TO_WORKER" pyspark
[..snip..]
In [1]: import os
In [2]: os.environ['PYTHONPATH']
Out[2]: '/mnt/spark/python/lib/py4j-0.8.2.1-src.zip:/mnt/spark/python/:DOESNT_GET_TO_WORKER'
In [3]: sc.parallelize([1]).map(lambda x: os.environ['PYTHONPATH']).first()
Out[3]: '/mnt/spark/python/lib/pyspark.zip:/mnt/spark/python/lib/py4j-0.8.2.1-src.zip:/mnt/spark/assembly/target/scala-2.10/spark-assembly-1.4.0-hadoop2.4.1.jar:/mnt/spark/sbin/../python/lib/py4j-0.8.2.1-src.zip:/mnt/spark/sbin/../python:/mnt/spark/sbin/../python/lib/py4j-0.8.2.1-src.zip:/mnt/spark/sbin/../python:'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment