Jie Fu bigaidream

🦄

Progressing

ML researcher, freestyle copy-paster

bigaidream / spark_ide.py

Last active January 14, 2018 08:30

To enable IDE (PyCharm) syntax support for Apache Spark, adopted from http://www.abisen.com/spark-from-ipython-notebook.html

	#!/public/spark-0.9.1/bin/pyspark

	import os
	import sys

	# Set the path for spark installation
	# this is the path where you have built spark using sbt/sbt assembly
	os.environ['SPARK_HOME'] = "/public/spark-0.9.1"
	# os.environ['SPARK_HOME'] = "/home/jie/d2/spark-0.9.1"
	# Append to PYTHONPATH so that pyspark could be found