Skip to content

Instantly share code, notes, and snippets.

@magigo
Last active October 2, 2021 15:29
Show Gist options
  • Save magigo/6c4cf2eff4e4b11fb867 to your computer and use it in GitHub Desktop.
Save magigo/6c4cf2eff4e4b11fb867 to your computer and use it in GitHub Desktop.
develop spark app with Pycharm
import os
import sys
import traceback
# Path for spark source folder
os.environ['SPARK_HOME']="~/Downloads/spark-1.2.0-bin-hadoop2.4"
# Append pyspark to Python Path
sys.path.append("~/Downloads/spark-1.2.0-bin-hadoop2.4/python/")
sys.path.append("~/Downloads/spark-1.2.0-bin-hadoop2.4/python/lib/py4j-0.8.2.1-src.zip")
try:
from pyspark import SparkContext
from pyspark import SparkConf
print ("Successfully imported Spark Modules")
except ImportError as e:
print ("Can not import Spark Modules {}".format(traceback.format_exc()))
sys.exit(1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment