Skip to content

Instantly share code, notes, and snippets.

@pphetra

pphetra/ex1.py

Created March 7, 2020 04:40
Show Gist options
  • Save pphetra/5eb673bc8e52ae9aadfb541c7be15249 to your computer and use it in GitHub Desktop.
Save pphetra/5eb673bc8e52ae9aadfb541c7be15249 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
ex1.py
covid060320 = spark\
.read\
.option("inferSchema", "true")\
.option("header", "true")\
.csv("03-06-2020.csv")
covid060320.take(10)
covid060320.sort("Deaths").explain()
spark.conf.set("spark.sql.shuffle.partitions", "5")
covid060320.sort("Deaths").explain()
Raw
ex1.scala
val covid060320 = spark
.read
.option("inferSchema", "true")
.option("header", "true")
.csv("03-06-2020.csv")
covid060320.take(10)
covid060320.sort("Deaths").explain()
spark.conf.set("spark.sql.shuffle.partitions", "5")
covid060320.sort("Deaths").explain()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment