Skip to content

Instantly share code, notes, and snippets.

@fspaolo

fspaolo/install_spark.md

Last active February 24, 2017 11:44
Show Gist options
  • Save fspaolo/53daaf85318ae89629dd to your computer and use it in GitHub Desktop.
Save fspaolo/53daaf85318ae89629dd to your computer and use it in GitHub Desktop.
Download ZIP
Install Apache Spark
Raw
install_spark.md

Install Apache Spark (OSX 10.6)

You need the package manager Homebrew (if not installed), see http://brew.sh/

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Get Spark 'Source Code':

http://spark.apache.org/downloads.html

Add to .bash_profile:

export SPARK_HOME=/path/to/spark/folder
source ~/.bash_profile

Install the Scala build tool and build Spark (took ~70 min):

cd $SPARK_HOME
brew install sbt
sbt assembly

If Error:

[error] (streaming-flume-sink/compile:compile) java.io.IOException: Cannot run program "/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/bin/javac": error=2, No such file or directory

Then link the bin folder where javac is to the path searched by sbt:

which javac
/usr/bin/javac
ln -s /usr/bin /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment