Skip to content

Instantly share code, notes, and snippets.

@dgadiraju

dgadiraju/spark-core-groupByKey-sorting.scala

Created September 5, 2019 14:46
Show Gist options
  • Save dgadiraju/9ce8013b22c8e21a0ecaf4b324ffa2f8 to your computer and use it in GitHub Desktop.
Save dgadiraju/9ce8013b22c8e21a0ecaf4b324ffa2f8 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark-core-groupByKey-sorting.scala
val products = sc.textFile("/public/retail_db/products")
val productsMap = products.
filter(p => p.split(",")(4) != "").
map(p => (p.split(",")(1).toInt, p))
productsMap.
groupByKey.
flatMap(p => {
p._2.toList.sortBy(k => k.split(",")(4).toFloat)(Ordering.Float.reverse)
}).
take(100).
foreach(println)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment