Skip to content

Instantly share code, notes, and snippets.

@sbatururimi

sbatururimi/pyspark_pass graphframes package.txt

Last active June 27, 2019 11:20
Show Gist options
  • Save sbatururimi/9e2f586faeac1f83bbcb1ec0d21834d6 to your computer and use it in GitHub Desktop.
Save sbatururimi/9e2f586faeac1f83bbcb1ec0d21834d6 to your computer and use it in GitHub Desktop.
Download ZIP
Install GraphFrames with Jupyter notebook
Raw
pyspark_pass graphframes package.txt
mkdir ~/jupyter
cd ~/jupyter
wget https://github.com/graphframes/graphframes/archive/release-0.6.0.zip
unzip release-0.6.0.zip
cd graphframes-release-0.6.0
build/sbt assembly
cd ..
# Copy necessary files to root level so we can start pyspark.
cp graphframes-release-0.6.0/target/scala-2.11/graphframes-assembly-0.6.0-spark2.3.jar .
cp -r graphframes-release-0.6.0/python/graphframes .
# Set environment to use Jupyter
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS=notebook
# Launch the jupyter server.
pyspark --jars graphframes-assembly-0.6.0-spark2.3.jar
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment