Created
November 18, 2020 10:43
-
-
Save shredder47/ce2f158a2a3907c0d264c5e9e4aab2fa to your computer and use it in GitHub Desktop.
Cloudera QuickStart VM , Spark Version Upgrade
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
java -version | |
sudo yum remove java | |
sudo yum install java-1.8.0-openjdk | |
source ~/.bash_profile | |
Download Spark 2.4.7 With Hadoop 2.6 (Tar) | |
Extract contents. | |
Move the contents of the folder to : | |
/usr/local/spark | |
Now, | |
Open: | |
/usr/bin/pyspark | |
/usr/bin/spark-shell | |
/usr/bin/spark-submit | |
and change the value for each files to | |
'exec /usr/local/spark/bin/pyspark "$@"' | |
'exec /usr/local/spark/bin/spark-shell "$@"' | |
'exec /usr/local/spark/bin/spark-submit "$@"' | |
Now try running spark to check the version |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment