Skip to content

Instantly share code, notes, and snippets.

@ashkrit

ashkrit/SparkDag.scala

Created September 24, 2018 00:20
Show Gist options
  • Save ashkrit/355b86a169e0e93fc463082d0ae89037 to your computer and use it in GitHub Desktop.
Save ashkrit/355b86a169e0e93fc463082d0ae89037 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
SparkDag.scala
val topXNumbers = randomNumbers
.filter(_ > 1000) //Stage 1
.map(value => (value, 1)) // Stage 1
.groupByKey() //Stage 2
.map(value => (value._1, value._2.sum)) //Stage 2
.sortBy(_._2, false) //Stage 3
.count() // Stage 3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment