Skip to content

Instantly share code, notes, and snippets.

@ianmilligan1

ianmilligan1/script-all-derivatives.scala

Last active December 21, 2017 21:33
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save ianmilligan1/559455fc19412ba4fbdb1b87a5659727 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save ianmilligan1/559455fc19412ba4fbdb1b87a5659727 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
script-all-derivatives.scala
import io.archivesunleashed.spark.matchbox.{ExtractDomain, ExtractLinks, RemoveHTML, RecordLoader, WriteGEXF}
import io.archivesunleashed.spark.rdd.RecordRDD._
val r = RecordLoader.loadArchives("/data/govt-info/*.gz", sc).keepValidPages().map(r => ExtractDomain(r.getUrl)).countItems().saveAsTextFile("/data/all-domains")
RecordLoader.loadArchives("/data/govt-info/*.gz", sc).keepValidPages().map(r => (r.getCrawlDate, r.getDomain, r.getUrl, RemoveHTML(r.getContentString))).saveAsTextFile("/data/all-text")
val links = RecordLoader.loadArchives("/data/govt-info/*.gz", sc).keepValidPages().map(r => (r.getCrawlDate, ExtractLinks(r.getUrl, r.getContentString))).flatMap(r => r._2.map(f => (r._1, ExtractDomain(f._1).replaceAll("^\\s*www\\.", ""), ExtractDomain(f._2).replaceAll("^\\s*www\\.", "")))).filter(r => r._2 != "" && r._3 != "").countItems().filter(r => r._2 > 5)
WriteGEXF(links, "/data/links-for-gephi.gexf")
sys.exit
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment