Ian Milligan ianmilligan1
ianmilligan1
/ fail-log3.txt
Created
January 4, 2018 18:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2018-01-04 18:00:34,213 [Executor task launch worker for task 119] ERROR Executor - Exception in task 119.0 in stage 0.0 (TID 119) | |
| java.lang.OutOfMemoryError: Java heap space | |
| at java.util.Arrays.copyOf(Arrays.java:3332) | |
| at java.lang.StringCoding.safeTrim(StringCoding.java:89) | |
| at java.lang.StringCoding.access$100(StringCoding.java:50) | |
| at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:154) | |
| at java.lang.StringCoding.decode(StringCoding.java:193) | |
| at java.lang.StringCoding.decode(StringCoding.java:254) | |
| at java.lang.String.<init>(String.java:546) | |
| at java.lang.String.<init>(String.java:566) |
ianmilligan1
/ fail-log2.txt
Created
January 4, 2018 16:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ubuntu@AUT-Troubleshooter:~/aut$ ./spark-2.1.1-bin-hadoop2.6/bin/spark-shell --driver-memory 45G --packages "io.archivesunleashed:aut:0.12.1" | |
| Ivy Default Cache set to: /home/ubuntu/.ivy2/cache | |
| The jars for the packages stored in: /home/ubuntu/.ivy2/jars | |
| :: loading settings :: url = jar:file:/home/ubuntu/aut/spark-2.1.1-bin-hadoop2.6/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml | |
| io.archivesunleashed#aut added as a dependency | |
| :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 | |
| confs: [default] | |
| found io.archivesunleashed#aut;0.12.1 in central | |
| found junit#junit;4.8.2 in central | |
| found org.scala-lang#scala-parser-combinators;2.11.0-M4 in central |
ianmilligan1
/ hs_err_pid2492.log
Created
January 4, 2018 04:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # There is insufficient memory for the Java Runtime Environment to continue. | |
| # Native memory allocation (mmap) failed to map 4703911936 bytes for committing reserved memory. | |
| # Possible reasons: | |
| # The system is out of physical RAM or swap space | |
| # In 32 bit mode, the process size limit was hit | |
| # Possible solutions: | |
| # Reduce memory load on the system | |
| # Increase physical memory or swap space | |
| # Check if swap backing store is full |
ianmilligan1
/ fail-log1.txt
Created
January 4, 2018 04:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ubuntu@AUT-Troubleshooter:~/aut$ ./spark-2.1.1-bin-hadoop2.6/bin/spark-shell --driver-memory 52G --packages "io.archivesunleashed:aut:0.12.1" | |
| Ivy Default Cache set to: /home/ubuntu/.ivy2/cache | |
| The jars for the packages stored in: /home/ubuntu/.ivy2/jars | |
| :: loading settings :: url = jar:file:/home/ubuntu/aut/spark-2.1.1-bin-hadoop2.6/jars/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml | |
| io.archivesunleashed#aut added as a dependency | |
| :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 | |
| confs: [default] | |
| found io.archivesunleashed#aut;0.12.1 in central | |
| found junit#junit;4.8.2 in central | |
| found org.scala-lang#scala-parser-combinators;2.11.0-M4 in central |
ianmilligan1
/ error-text-extract.log
Created
December 24, 2017 21:17
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [Stage 5:===> (538 + 8) / 8073]2017-12-24 17:39:02,871 [dispatcher-event-loop-4] WARN HeartbeatReceiver - Removing executor driver with no recent heartbeats: 236195 ms exceeds timeout 120000 ms | |
| [Stage 5:===> (538 + 8) / 8073]2017-12-24 17:39:02,873 [dispatcher-event-loop-4] ERROR TaskSchedulerImpl - Lost executor driver on localhost: Executor heartbeat timed out after 236195 ms | |
| 2017-12-24 17:39:02,891 [dispatcher-event-loop-4] WARN TaskSetManager - Lost task 538.0 in stage 5.0 (TID 25793, localhost, executor driver): ExecutorLostFailure (executor driver exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 236195 ms | |
| 2017-12-24 17:39:02,894 [dispatcher-event-loop-4] ERROR TaskSetManager - Task 538 in stage 5.0 failed 1 times; aborting job | |
| 2017-12-24 17:39:02,898 [dispatcher-event-loop-4] WARN TaskSetManager - Lost task 541.0 in stage 5.0 (TID 25796, localhost, executor driver): E |
ianmilligan1
/ error-arc-links
Created
December 15, 2017 16:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [Stage 0:> (19 + 8) / 2441]2017-12-15 16:32:17,654 [Executor task launch worker for task 26] ERROR Executor - Exception in task 26.0 in stage 0.0 (TID 26) | |
| java.lang.StackOverflowError | |
| at org.jsoup.parser.HtmlTreeBuilderState$7.process(HtmlTreeBuilderState.java:279) | |
| at org.jsoup.parser.HtmlTreeBuilder.process(HtmlTreeBuilder.java:142) | |
| at org.jsoup.parser.HtmlTreeBuilderState$15.anythingElse(HtmlTreeBuilderState.java:1207) | |
| at org.jsoup.parser.HtmlTreeBuilderState$15.process(HtmlTreeBuilderState.java:1201) | |
| at org.jsoup.parser.HtmlTreeBuilder.process(HtmlTreeBuilder.java:137) | |
| at org.jsoup.parser.TreeBuilder.processStartTag(TreeBuilder.java:66) | |
| at org.jsoup.parser.HtmlTreeBuilderState$7.process(HtmlTreeBuilderState.java:707) |
ianmilligan1
/ script-all-derivatives.scala
Last active
December 21, 2017 21:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import io.archivesunleashed.spark.matchbox.{ExtractDomain, ExtractLinks, RemoveHTML, RecordLoader, WriteGEXF} | |
| import io.archivesunleashed.spark.rdd.RecordRDD._ | |
| val r = RecordLoader.loadArchives("/data/govt-info/*.gz", sc).keepValidPages().map(r => ExtractDomain(r.getUrl)).countItems().saveAsTextFile("/data/all-domains") | |
| RecordLoader.loadArchives("/data/govt-info/*.gz", sc).keepValidPages().map(r => (r.getCrawlDate, r.getDomain, r.getUrl, RemoveHTML(r.getContentString))).saveAsTextFile("/data/all-text") | |
| val links = RecordLoader.loadArchives("/data/govt-info/*.gz", sc).keepValidPages().map(r => (r.getCrawlDate, ExtractLinks(r.getUrl, r.getContentString))).flatMap(r => r._2.map(f => (r._1, ExtractDomain(f._1).replaceAll("^\\s*www\\.", ""), ExtractDomain(f._2).replaceAll("^\\s*www\\.", "")))).filter(r => r._2 != "" && r._3 != "").countItems().filter(r => r._2 > 5) | |
| WriteGEXF(links, "/data/links-for-gephi.gexf") | |
| sys.exit |
ianmilligan1
/ script-gephi.scala
Last active
December 14, 2017 20:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import io.archivesunleashed.spark.matchbox.{ExtractDomain, ExtractLinks, RecordLoader, WriteGEXF} | |
| import io.archivesunleashed.spark.rdd.RecordRDD._ | |
| val links = RecordLoader.loadArchives("/data/cpp/*.gz", sc).keepValidPages().map(r => (r.getCrawlDate, ExtractLinks(r.getUrl, r.getContentString))).flatMap(r => r._2.map(f => (r._1, ExtractDomain(f._1).replaceAll("^\\s*www\\.", ""), ExtractDomain(f._2).replaceAll("^\\s*www\\.", "")))).filter(r => r._2 != "" && r._3 != "").countItems().filter(r => r._2 > 5) | |
| WriteGEXF(links, "/home/ubuntu/links-for-gephi.gdf") | |
| sys.exit |
ianmilligan1
/ script.scala
Last active
December 13, 2017 19:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import io.archivesunleashed.spark.matchbox._ | |
| import io.archivesunleashed.spark.rdd.RecordRDD._ | |
| val r = RecordLoader.loadArchives("/home/ubuntu/data/*.gz", sc) | |
| .keepValidPages() | |
| .map(r => r.getUrl) | |
| .saveAsTextFile("/home/ubuntu/data/output") | |
| sys.exit |
ianmilligan1
/ tweet.json
Created
December 13, 2017 01:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "contributors": null, | |
| "truncated": false, | |
| "text": "Give it your all now, @ThomasMulcair! You do you. #elxn42 #cdnpoli", | |
| "is_quote_status": false, | |
| "in_reply_to_status_id": null, | |
| "id": 656312151578189800, | |
| "favorite_count": 0, | |
| "source": "<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>", | |
| "retweeted": false, |