Skip to content

Instantly share code, notes, and snippets.

@zouzias

zouzias/compileSpark_2.11.sh

Created May 30, 2016 14:39
Show Gist options
  • Save zouzias/48b2eb565da1b5d2f04c41344338b915 to your computer and use it in GitHub Desktop.
Save zouzias/48b2eb565da1b5d2f04c41344338b915 to your computer and use it in GitHub Desktop.
Download ZIP
How to compile Spark with 2.11 support
Raw
compileSpark_2.11.sh
git clean -ffxd
SPARK_VERSION="1.5.1"
git checkout -f "v$SPARK_VERSION" --
sh dev/change-scala-version.sh 2.11
export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m -XX:+CMSClassUnloadingEnabled"
mvn clean
./make-distribution.sh --tgz --skip-java-test -Pscala-2.11 -Pyarn -Phadoop-2.6 -DskipTests -Dspark.version=$SPARK_VERSION -Dscala.version=2.11.7 -DautoVersionSubmodules=true -U -Djline.version=2.13 -Djline.groupid=jline
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment