Ryan Williams ryan-williams
🚆
ryan-williams
/ 1.4.0 - success
Created
September 27, 2015 17:43
"Provider org.apache.hadoop.fs.s3.S3FileSystem not found" error at different Spark versions - caused by stray META-INF in the directory spark-shell was launched from.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ spark-select 1.4.0 | |
| $ $SPARK_HOME/bin/spark-shell | |
| log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). | |
| log4j:WARN Please initialize the log4j system properly. | |
| log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. | |
| Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties | |
| 15/09/26 05:51:04 INFO SecurityManager: Changing view acls to: willir31 | |
| 15/09/26 05:51:04 INFO SecurityManager: Changing modify acls to: willir31 | |
| 15/09/26 05:51:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(willir31); users with modify permissions: Set(willir31) | |
| 15/09/26 05:51:04 INFO HttpServer: Starting HTTP Server |
ryan-williams
/ blaze-template.js
Created
October 4, 2015 23:13
React component for rendering Blaze templates inside other React components. Adapted from https://github.com/jedwards1211/meteor-webpack-react/blob/a4cf9be698a3ded97ffa4bb53b11ede0c39d364e/app/components/BlazeTemplate.js and https://forums.meteor.com/t/calling-meteor-template-from-react-components/8704
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| BlazeTemplate = React.createClass({ | |
| propTypes: { | |
| template: React.PropTypes.any.isRequired, | |
| component: React.PropTypes.any, | |
| }, | |
| getDefaultProps() { | |
| return { | |
| component: 'div' |
ryan-williams
/ blaze-template.js
Created
October 4, 2015 23:13
React component for rendering Blaze templates inside other React components. Adapted from https://github.com/jedwards1211/meteor-webpack-react/blob/a4cf9be698a3ded97ffa4bb53b11ede0c39d364e/app/components/BlazeTemplate.js and https://forums.meteor.com/t/calling-meteor-template-from-react-components/8704
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| BlazeTemplate = React.createClass({ | |
| propTypes: { | |
| template: React.PropTypes.any.isRequired, | |
| component: React.PropTypes.any, | |
| }, | |
| getDefaultProps() { | |
| return { | |
| component: 'div' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_001.C6G1RANXX.fastq | |
| /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_002.C6G1RANXX.fastq | |
| /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_003.C6G1RANXX.fastq | |
| /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_004.C6G1RANXX.fastq | |
| /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_005.C6G1RANXX.fastq | |
| /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_006.C6G1RANXX.fastq | |
| /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_007.C6G1RANXX.fastq | |
| /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 3.9 G /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_001.C6G1RANXX.fastq | |
| 3.9 G /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_002.C6G1RANXX.fastq | |
| 3.9 G /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_003.C6G1RANXX.fastq | |
| 3.9 G /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_004.C6G1RANXX.fastq | |
| 3.9 G /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_005.C6G1RANXX.fastq | |
| 3.9 G /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_006.C6G1RANXX.fastq | |
| 3.4 G /datasets/martignetti_ovarian/189/Illumina_DNA/PT189_11_13/Raw/DNA.IlluminaHiSeq2500.WES/PT189_11_13_AGAGTCAA_L006_R1_007.C6G1RANXX.fastq | |
| 3.9 G /datasets/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {"Event":"SparkListenerLogStart","Spark Version":"1.5.2"} | |
| {"Event":"SparkListenerBlockManagerAdded","Block Manager ID":{"Executor ID":"driver","Host":"172.29.46.15","Port":53169},"Maximum Memory":4782387363,"Timestamp":1449005263345} | |
| {"Event":"SparkListenerEnvironmentUpdate","JVM Information":{"Java Home":"/demeter/users/willir31/jdk1.7.0_79/jre","Java Version":"1.7.0_79 (Oracle Corporation)","Scala Version":"version 2.10.4"},"Spark Properties":{"spark.serializer":"org.apache.spark.serializer.KryoSerializer","spark.speculation":"true","spark.driver.host":"172.29.46.15","spark.eventLog.enabled":"true","spark.shuffle.manager":"SORT","spark.driver.port":"46372","spark.shuffle.service.enabled":"true","spark.kryo.registrator":"org.hammerlab.pageant.kryo.PageantKryoRegistrar","spark.repl.class.uri":"http://172.29.46.15:44755","spark.jars":"file:/demeter/users/willir31/spark-notebook-dad0ba2-scala-2.10.4-spark-1.5.2-hadoop-2.6.0/,file:/demeter/users/willir31/spark-notebook-dad0ba2-scala-2.10.4-spark-1.5.2-hadoop-2.6 |
ryan-williams
/ job.108
Created
January 23, 2016 21:22
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2015-12-04 18:03:53,360 INFO [Remote-akka.actor.default-dispatcher-3] (org.apache.spark.SparkContext) - Starting job: zipWithIndex at PDC3.scala:324 | |
| 2015-12-04 18:03:53,408 INFO [task-result-getter-2] (org.apache.spark.scheduler.TaskSetManager) - Ignoring task-finished event for 1171.1 in stage 6843.0 because task 1171 has already completed successfully | |
| 2015-12-04 18:03:53,439 INFO [task-result-getter-3] (org.apache.spark.scheduler.TaskSetManager) - Ignoring task-finished event for 1855.0 in stage 6843.0 because task 1855 has already completed successfully | |
| 2015-12-04 18:03:53,473 INFO [dag-scheduler-event-loop] (org.apache.spark.MapOutputTrackerMaster) - Size of output statuses for shuffle 0 is 53989 bytes | |
| 2015-12-04 18:03:53,575 INFO [task-result-getter-1] (org.apache.spark.scheduler.TaskSetManager) - Ignoring task-finished event for 98.0 in stage 6843.0 because task 98 has already completed successfully | |
| 2015-12-04 18:03:53,581 INFO [task-result-getter-0] (org.apache.spark.scheduler.TaskSetManager) - I |
ryan-williams
/ gist:bf2989d1aa18cb998eaa
Created
February 11, 2016 18:57
enable full stack traces in scalatest in maven: <stdout>F</stdout>
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <plugin> | |
| <groupId>org.scalatest</groupId> | |
| <artifactId>scalatest-maven-plugin</artifactId> | |
| <configuration> | |
| <reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory> | |
| <junitxml>.</junitxml> | |
| <filereports>ADAMTestSuite.txt</filereports> | |
| <!-- | |
| As explained here: http://stackoverflow.com/questions/1660441/java-flag-to-enable-extended-serialization-debugging-info | |
| The second option allows us better debugging for serialization-based errors. |
ryan-williams
/ apple tv #1, part 1
Last active
March 5, 2016 01:03
Ping times to google.com while turning airplay on and off.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| $ ping google.com | |
| PING google.com (74.125.29.101): 56 data bytes | |
| 64 bytes from 74.125.29.101: icmp_seq=0 ttl=40 time=26.996 ms | |
| 64 bytes from 74.125.29.101: icmp_seq=1 ttl=40 time=29.221 ms | |
| 64 bytes from 74.125.29.101: icmp_seq=2 ttl=40 time=28.450 ms | |
| 64 bytes from 74.125.29.101: icmp_seq=3 ttl=40 time=36.275 ms | |
| 64 bytes from 74.125.29.101: icmp_seq=4 ttl=40 time=34.081 ms | |
| 64 bytes from 74.125.29.101: icmp_seq=5 ttl=40 time=30.861 ms | |
| 64 bytes from 74.125.29.101: icmp_seq=6 ttl=40 time=32.789 ms | |
| 64 bytes from 74.125.29.101: icmp_seq=7 ttl=40 time=29.107 ms |
ryan-williams
/ test.sh
Last active
March 31, 2016 14:44
How to make perl match a run of spaces and newlines?
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # File "bar"'s whitespace consists only of spaces and newlines. | |
| $ cat bar | |
| 1, 2, | |
| 3, | |
| 4 | |
| # All attempts to replace consecutive runs of whitespace (spaces and newlines) with 'S' | |
| # result in \n being replaced by one 'S', and spaces that follow it replaced by another 'S'. | |
| # Desired output: 1,S2,S3,S4S | |
| # Actual output: 1,S2,SS3,S4S |