Skip to content

Instantly share code, notes, and snippets.

@habedi
Last active February 12, 2020 21:18
Show Gist options
  • Save habedi/24ee508c348c37f9d12dc522f6afe6cb to your computer and use it in GitHub Desktop.
Save habedi/24ee508c348c37f9d12dc522f6afe6cb to your computer and use it in GitHub Desktop.
Simple commands to download and extract Apache Spark's pre-built binaries from its websites
## run the following commands in BASH
cd # let's get back to your user's home directory
wget -c https://www-us.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz # this will download spark
tar xvfz spark-2.4.5-bin-hadoop2.7.tgz # this will extract the downloaded file to current directory
mv spark-2.4.5-bin-hadoop2.7 spark # renaming the extarcted folder to "spark"
# appending the JAVA_HOME and SPARK_HOME environement variables to end of your BASH startup script
# we are assuming that our JRE 8 is installed in "/usr/lib/jvm/java-1.8.0-openjdk-amd64"
cat >> .bashrc <<'EOF'
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH
export SPARK_HOME="/home/$USER/spark"
export PATH=$SPARK_HOME/sbin:$SPARK_HOME/bin:$PATH
EOF
# let's restart our terminal
bash
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment