Skip to content

Instantly share code, notes, and snippets.

@juriad
Created March 15, 2017 13:52
Show Gist options
  • Select an option

  • Save juriad/f8929117048bd86330d5c645820891c5 to your computer and use it in GitHub Desktop.

Select an option

Save juriad/f8929117048bd86330d5c645820891c5 to your computer and use it in GitHub Desktop.
Create a histogram of people's balances using Spark
class CDA {
String creditor;
String debtor;
int amount;
}
public List<Tuple2<Integer, Integer>> createHistogram(JavaRDD<CDA> inputRDD) {
return inputRDD
.flatMapToPair(cda -> Arrays.asList(
new Tuple2<>(cda.creditor, cda.amount),
new Tuple2<>(cda.debtor, -cda.amount)))
.reduceByKey(Integer::sum)
.mapToPair(pa -> new Tuple2<>(pa._2 / 1000, 1))
.reduceByKey(Integer::sum)
.sortByKey(true, 1)
.collect();
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment