Skip to content

Instantly share code, notes, and snippets.

@solidpple

solidpple/PageRankAlgorithm.scala

Created July 26, 2016 00:09
Show Gist options
  • Save solidpple/ef7709533ec86207b4775745bba75d5d to your computer and use it in GitHub Desktop.
Save solidpple/ef7709533ec86207b4775745bba75d5d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
PageRankAlgorithm.scala
val links = sc.objectFile[(String, Seq[String])]("links").partitionBy(new HashPartitioner(100)).persist()
val ranks = links.mapValues(v => 1.0)
for (i <- 0 until 10) {
val contributions = links.join(ranks).flatMap {
case (pageId, (link, rank)) =>
links.map(dest => (dest, rank / links.size))
}
rank = contributions.reduceByKey((x, y) => x + y).mapValues(v => 0.15 + 0.85*v)
}
ranks.saveAsTextFile("ranks")
@victron777
Copy link

Hi, I would appreciate if anyone provide me the ObjectFile that this script is running. Thanks in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment