Skip to content

Instantly share code, notes, and snippets.

@zhuqling

zhuqling/macOS X 安装pyspark + VSCode IDE环境.md

Last active January 10, 2020 07:31
Show Gist options
  • Save zhuqling/8996aa36ee86470a5b58e22523bf38b7 to your computer and use it in GitHub Desktop.
Save zhuqling/8996aa36ee86470a5b58e22523bf38b7 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
macOS X 安装pyspark + VSCode IDE环境.md

安装pyspark + VSCode IDE环境

环境macos 10.14

  • 安装Java 8，系统默认是Java 10，需到oracle官网下载并手动安装
  • 安装scala, brew install scale
  • python版本，建议python 2.7
  • 安装spark, brew install apache-spark
  • 调整配置文件vi ~/.bash_profile，注意版本号调整到正常
export SPARK_HOME=/usr/local/Cellar/apache-spark/2.4.4/libexec
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
export PATH=$SPARK_HOME/python:$PATH
  • 重新加载配置文件 source ~/.bash_profile
  • 一定要点击左下角切换python版本，为安装了pyspark的版本
  • 打开.py文件，点击右上角的运行按钮即可
  • 点击调试，自动生成launch.json文件，便可以断点调试

可选步骤

  • VSCode 安装Spark & Hive Tools
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment