Skip to content

Instantly share code, notes, and snippets.

@mleuthold

mleuthold/prepare-pypspark-env.sh

Last active October 4, 2023 19:42
Show Gist options
  • Save mleuthold/5067c2b3bad158bb8a8822b8cf21b1e8 to your computer and use it in GitHub Desktop.
Save mleuthold/5067c2b3bad158bb8a8822b8cf21b1e8 to your computer and use it in GitHub Desktop.
Download ZIP
Use PySpark 3.0.0 with pyenv
Raw
prepare-pypspark-env.sh
pyenv install 3.8.1
pyenv virtualenv 3.8.1 spark3
pyenv shell spark3
pip install pyspark
MY_PYSPARK_LOCATION=$(pip show pyspark | grep -e "Location" | cut -d ' ' -f 2)
# /home/marleu/.pyenv/versions/3.8.1/envs/spark3/lib/python3.8/site-packages
MY_PYSPARK_NAME=$(pip show pyspark | grep -e "Name" | cut -d ' ' -f 2)
# pyspark
export SPARK_HOME="$MY_PYSPARK_LOCATION/$MY_PYSPARK_NAME"
export PYSPARK_PYTHON=$(pyenv which python)
pyspark
# JAVA 11
/usr/lib/jvm/java-11-openjdk-amd64/
# JAVA 8
/usr/lib/jvm/java-8-openjdk-amd64/
export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64/"; spark-shell --version
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment