John Ferguson InvisibleTech
InvisibleTech
/ joiner.scala
Created
February 5, 2016 01:34
Apache Spark Example - take two colums, join them and duplicate the other.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // Need to :paste this into Spark Shell to see it work. | |
| // | |
| // Load up the columns | |
| val alpha = sc.parallelize(List("a", "b", "c", "d")) | |
| val nums = sc.parallelize(List(1, 2, 3, 4)) | |
| // Key them by index | |
| val alphaK = alpha.zipWithIndex.map(t => (t._2, t._1)) | |
| val numsK = nums.zipWithIndex.map(t => (t._2, t._1)) |
OlderNewer