Skip to content

Instantly share code, notes, and snippets.

@tmcgrath

tmcgrath/SparkSQL with json example

Created December 8, 2016 13:43
Show Gist options
  • Save tmcgrath/d40435667d261fbc2a593dc331cb2c8a to your computer and use it in GitHub Desktop.
Save tmcgrath/d40435667d261fbc2a593dc331cb2c8a to your computer and use it in GitHub Desktop.
Download ZIP
SparkSQL with json file example
Raw
SparkSQL with json example
// download sample json
http://bit.ly/2gY39Ay
// start spark-shell in same directory as where customers.json
// is downloaded
val customers = sqlContext.jsonFile("customers.json")
// register a temp table
customers.registerTempTable("customers")
// create dataframe with sql
val firstCityState = sqlContext.sql("SELECT first_name, address.city, address.state FROM customers")
firstCityState.show
firstCityState.collect.foreach(println)
// etc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment