Skip to content

Instantly share code, notes, and snippets.

@shredder47
Created November 18, 2020 10:43
Show Gist options
  • Save shredder47/ce2f158a2a3907c0d264c5e9e4aab2fa to your computer and use it in GitHub Desktop.
Save shredder47/ce2f158a2a3907c0d264c5e9e4aab2fa to your computer and use it in GitHub Desktop.
Cloudera QuickStart VM , Spark Version Upgrade
java -version
sudo yum remove java
sudo yum install java-1.8.0-openjdk
source ~/.bash_profile
Download Spark 2.4.7 With Hadoop 2.6 (Tar)
Extract contents.
Move the contents of the folder to :
/usr/local/spark
Now,
Open:
/usr/bin/pyspark
/usr/bin/spark-shell
/usr/bin/spark-submit
and change the value for each files to
'exec /usr/local/spark/bin/pyspark "$@"'
'exec /usr/local/spark/bin/spark-shell "$@"'
'exec /usr/local/spark/bin/spark-submit "$@"'
Now try running spark to check the version
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment