Skip to content

Instantly share code, notes, and snippets.

@tromika

tromika/test.py

Created December 18, 2017 11:00
Show Gist options
  • Save tromika/1cda392242fdd66befe7970d80380216 to your computer and use it in GitHub Desktop.
Save tromika/1cda392242fdd66befe7970d80380216 to your computer and use it in GitHub Desktop.
Download ZIP
Spark on Windows
Raw
test.py
from pyspark import SparkContext, SparkConf
import pyspark
import os
os.environ["JAVA_HOME"] = "C:\Program Files\Java\jre1.8.0_151"
os.environ['SPARK_HOME']="C:\spark-2.2.1-bin-hadoop2.7"
os.environ['HADOOP_HOME']="C:\hadoop-2.7.1"
config = SparkConf().setMaster("local[8]").set("spark.executor.memory","8g")
spark = pyspark.sql.SparkSession.builder.config(conf=config).appName('test').getOrCreate()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment