Created
August 7, 2017 21:08
-
-
Save IgorBerman/52d18126f29f995bca2d3af5af293258 to your computer and use it in GitHub Desktop.
intellij osx python spark
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Supposing you've installed spark with brew install [email protected] | |
| it will place it into /usr/local/opt/[email protected] | |
| to use spark locally from intellij from your script define following: | |
| 1. environment variables: | |
| PYSPARK_PYTHON=/Users/igorberm/.runtimes/Python34/bin/python3 | |
| PYTHONPATH=/usr/local/opt/[email protected]/libexec/python/lib/pyspark.zip:/usr/local/opt/[email protected]/libexec/python/lib/py4j-0.9-src.zip | |
| PYTHONUNBUFFERED=1 | |
| SPARK_HOME=/usr/local/opt/[email protected]/libexec | |
| 2. Use specified interpreter which will point to python sdk interpreter | |
| 3. Define sdk + interpreter from module setting: | |
| Add sdk with two classpathes | |
| /usr/local/opt/[email protected]/libexec/python/lib/pyspark.zip | |
| /usr/local/opt/[email protected]/libexec/python/lib/py4j-0.9-src.zip | |
| 4. In global libraries add both of them as Sources |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment