The reality of any hands-on workshop is that things will break. We've tried our best to provide a robust environment that can let you walk through the basics of the Archives Unleashed Toolkit alongside us.
If you have any questions, let us know!
Ian Milligan ianmilligan1
ianmilligan1
/ gexf-error.log
Created
November 16, 2018 23:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| scala> :paste | |
| // Entering paste mode (ctrl-D to finish) | |
| import io.archivesunleashed._ | |
| import io.archivesunleashed.app._ | |
| import io.archivesunleashed.matchbox._ | |
| val links = RecordLoader.loadArchives("/mnt/vol1/data_sets/geocities/warcs/indexed/GEOCITIES-20091029185858-00184-crawling08.us.archive.org.warc.gz", sc) | |
| .keepValidPages() | |
| .map(r => (r.getCrawlDate, ExtractLinks(r.getUrl, r.getContentString))) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| scala> :paste | |
| // Entering paste mode (ctrl-D to finish) | |
| import io.archivesunleashed._ | |
| import io.archivesunleashed.app._ | |
| import io.archivesunleashed.matchbox._ | |
| sc.setLogLevel("INFO") | |
| RecordLoader.loadArchives("/mnt/vol1/data_sets/aut_debug/issue-499/", sc).keepValidPages().map(r => ExtractDomain(r.getUrl)).countItems().saveAsTextFile("/mnt/vol1/data_sets/aut_debug/issue-499/results/domains") | |
| RecordLoader.loadArchives("/mnt/vol1/data_sets/aut_debug/issue-499/", sc).keepValidPages().map(r => (r.getCrawlDate, r.getDomain, r.getUrl, RemoveHTML(r.getContentString))).saveAsTextFile("/mnt/vol1/data_sets/aut_debug/issue-499/results/test") |
ianmilligan1
/ graphml
Created
June 7, 2018 22:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 4.0K ./739/10218/8/derivatives/gephi/10218-gephi.graphml | |
| 4.0K ./739/3951/8/derivatives/gephi/3951-gephi.graphml | |
| 4.0K ./739/6178/8/derivatives/gephi/6178-gephi.graphml | |
| 4.0K ./75/5715/2/derivatives/gephi/5715-gephi.graphml | |
| 4.0K ./990/6275/1/derivatives/gephi/6275-gephi.graphml | |
| 4.0K ./990/6280/1/derivatives/gephi/6280-gephi.graphml | |
| 4.0K ./990/6566/1/derivatives/gephi/6566-gephi.graphml | |
| 4.0K ./990/7515/18/derivatives/gephi/7515-gephi.graphml | |
| 4.0K ./990/7515/9/derivatives/gephi/7515-gephi.graphml | |
| 4.0K ./990/8155/1/derivatives/gephi/8155-gephi.graphml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| scala> :paste | |
| // Entering paste mode (ctrl-D to finish) | |
| import io.archivesunleashed._ | |
| import io.archivesunleashed.matchbox._ | |
| val r = RecordLoader.loadArchives("/mnt/vol1/data_sets/auk_datasets/4532/warcs/*.gz", sc) | |
| .keepValidPages() | |
| .map(r => (r.getCrawlDate, r.getDomain, r.getUrl, RemoveHTML(r.getContentString))) | |
| .saveAsTextFile("/mnt/vol1/derivative_data/auk-test/text") |
ianmilligan1
/ image-crash.log
Created
May 17, 2018 15:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [Stage 0:> (0 + 12) / 1147]2018-05-17 11:29:26 ERROR Executor:91 - Exception in task 8.0 in stage 0.0 (TID 8) | |
| java.io.IOException: Invalid image dimensions | |
| at io.archivesunleashed.matchbox.ExtractImageDetails$.apply(ExtractImageDetails.scala:62) | |
| at io.archivesunleashed.package$WARecordRDD$$anonfun$9.apply(package.scala:147) | |
| at io.archivesunleashed.package$WARecordRDD$$anonfun$9.apply(package.scala:146) | |
| at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) | |
| at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) | |
| at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) | |
| at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) | |
| at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) |
ianmilligan1
/ stopwords-auk.txt
Last active
March 12, 2018 15:11
Adapted from https://github.com/sgsinclair/trombone
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| a | |
| b | |
| c | |
| d | |
| e | |
| f | |
| g | |
| h | |
| i | |
| g |
ianmilligan1
/ big-error-log-domains.log
Created
January 30, 2018 20:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2018-01-30 16:49:07,894 [main] WARN NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
| 2018-01-30 16:49:07,963 [main] WARN Utils - Your hostname, indexer resolves to a loopback address: 127.0.0.1; using 192.168.32.19 instead (on interface ens3) | |
| 2018-01-30 16:49:07,963 [main] WARN Utils - Set SPARK_LOCAL_IP if you need to bind to another address | |
| 2018-01-30 16:49:13,243 [main] WARN ObjectStore - Failed to get database global_temp, returning NoSuchObjectException | |
| Spark context Web UI available at http://192.168.32.19:4040 | |
| Spark context available as 'sc' (master = local[12], app id = local-1517330948464). | |
| Spark session available as 'spark'. | |
| Loading university-of-alberta-websites.scala... | |
| import io.archivesunleashed.spark.matchbox.{ExtractDomain, ExtractLinks, RemoveHTML, RecordLoader, WriteGEXF} | |
| import io.archivesunleashed.spark.rdd.RecordRDD._ |
ianmilligan1
/ error-dump.log
Created
January 30, 2018 16:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ubuntu@indexer:~/aut-plumbing/Scripts$ cat university-of-alberta-websites.log | |
| 2018-01-30 16:11:08,030 [main] WARN NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable | |
| 2018-01-30 16:11:08,106 [main] WARN Utils - Your hostname, indexer resolves to a loopback address: 127.0.0.1; using 192.168.32.19 instead (on interface ens3) | |
| 2018-01-30 16:11:08,106 [main] WARN Utils - Set SPARK_LOCAL_IP if you need to bind to another address | |
| 2018-01-30 16:11:13,858 [main] WARN ObjectStore - Failed to get database global_temp, returning NoSuchObjectException | |
| Spark context Web UI available at http://192.168.32.19:4040 | |
| Spark context available as 'sc' (master = local[12], app id = local-1517328668626). | |
| Spark session available as 'spark'. | |
| Loading university-of-alberta-websites.scala... | |
| import io.archivesunleashed.spark.matchbox.{ExtractDomain, ExtractLinks, RemoveHTML, RecordLoader, WriteGEXF} |
ianmilligan1
/ docker-lesson-plan.md
Last active
January 24, 2018 20:38
ianmilligan1
/ spark-log-fail.txt
Created
January 5, 2018 13:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"Event":"SparkListenerLogStart","Spark Version":"2.1.1"} | |
| {"Event":"SparkListenerExecutorAdded","Timestamp":1515121514432,"Executor ID":"driver","Executor Info":{"Host":"localhost","Total Cores":16,"Log Urls":{}}} | |
| {"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"driver","Host":"10.0.3.4","Port":36183},"Maximum Memory":25581060096,"Timestamp":1515121514460} | |
| {"Event":"SparkListenerEnvironmentUpdate","JVM Information":{"Java Home":"/usr/lib/jvm/java-8-openjdk-amd64/jre","Java Version":"1.8.0_151 (Oracle Corporation)","Scala Version":"version 2.11.8"},"Spark Properties":{"spark.executor.extraJavaOptions":"-XX:+PrintGCDetails -XX:+PrintGCTimeStamps","spark.driver.host":"10.0.3.4","spark.eventLog.enabled":"true","spark.driver.port":"36135","spark.repl.class.uri":"spark://10.0.3.4:36135/classes","spark.jars":"file:/home/ubuntu/.ivy2/jars/io.archivesunleashed_aut-0.12.1.jar,file:/home/ubuntu/.ivy2/jars/junit_junit-4.8.2.jar,file:/home/ubuntu/.ivy2/jars/org.scala-lang_scala-parser-combinato |