Skip to content

Instantly share code, notes, and snippets.

@WanderingStar

WanderingStar/gist:aa22e166ae19579d2fa2d3767e8598f5

Created July 25, 2017 03:32
Show Gist options
  • Save WanderingStar/aa22e166ae19579d2fa2d3767e8598f5 to your computer and use it in GitHub Desktop.
Save WanderingStar/aa22e166ae19579d2fa2d3767e8598f5 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
df1 = spark.createDataFrame(rdd1).toDF("id", "val")
df1.createOrReplaceTempView("table1")
df2 = spark.createDataFrame(rdd2).toDF("id", "val")
df2.createOrReplaceTempView("table2")
def callable_ssql_timeit():
query="SELECT * FROM table1 LEFT JOIN table2 ON table1.id=table2.id"
spark.sql(query)
print(timeit.timeit(callable_ssql_timeit, number=100))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment