Jason Amster jamster
jamster
/ gist:513129
Created
August 7, 2010 20:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| module MyModule | |
| def print_hello | |
| puts "hello" | |
| end | |
| end | |
| class MyClass | |
| include MyModule | |
| end |
jamster
/ gist:514470
Created
August 8, 2010 20:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [jasonamster] lib [master*]$ hadoop jar build/wordcount.jar data/url+page.200.txt output local | |
| Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/PlatformName | |
| Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.util.PlatformName | |
| at java.net.URLClassLoader$1.run(URLClassLoader.java:202) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at java.net.URLClassLoader.findClass(URLClassLoader.java:190) | |
| at java.lang.ClassLoader.loadClass(ClassLoader.java:307) | |
| at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) | |
| at java.lang.ClassLoader.loadClass(ClassLoader.java:248) | |
| Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/util/RunJar |
jamster
/ gist:514550
Created
August 8, 2010 21:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| META-INF/ | |
| META-INF/MANIFEST.MF | |
| lib/ | |
| wordcount/ | |
| lib/cascading-core-1.2-wip-dev.jar | |
| log4j.properties | |
| wordcount/Main$ImportCrawlDataAssembly.class | |
| wordcount/Main$WordCountSplitAssembly.class | |
| wordcount/Main.class | |
| lib/cascading-1.1.2-wip-dev.jar |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [jasonamster] wordcount [master*]$ java -cp build/wordcount.jar:/usr/local/src/hadoop-0.20.2/hadoop-0.20.2-core.jar wordcount.Main data/url+page.200.txt output local | |
| Exception in thread "main" java.lang.NoClassDefFoundError: cascading/scheme/Scheme | |
| Caused by: java.lang.ClassNotFoundException: cascading.scheme.Scheme | |
| at java.net.URLClassLoader$1.run(URLClassLoader.java:202) | |
| at java.security.AccessController.doPrivileged(Native Method) | |
| at java.net.URLClassLoader.findClass(URLClassLoader.java:190) | |
| at java.lang.ClassLoader.loadClass(ClassLoader.java:307) | |
| at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) | |
| at java.lang.ClassLoader.loadClass(ClassLoader.java:248) |
jamster
/ gist:514567
Created
August 8, 2010 21:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [jasonamster] wordcount [master*]$ java -cp build/wordcount.jar:/usr/local/src/hadoop-0.20.2/hadoop-0.20.2-core.jar:/usr/local/src/hadoop-0.20.2/lib/log4j-1.2.15.jar:/projects/cascading/build/cascading-1.1.2-wip-dev.jar:/projects/cascading/build/cascading-core-1.1.2-wip-dev.jar:/projects/cascading/build/cascading-test-1.1.2-wip-dev.jar:/projects/cascading/build/cascading-xml-1.1.2-wip-dev.jar:/projects/cascading/lib/janino-2.5.16.jar:/projects/cascading/lib/jgrapht-jdk1.6.jar:/usr/local/src/hadoop-0.20.2/libcommons-cli-1.2.jar:/usr/local/src/hadoop-0.20.2/libcommons-codec-1.3.jar:/usr/local/src/hadoop-0.20.2/libcommons-el-1.0.jar:/usr/local/src/hadoop-0.20.2/libcommons-httpclient-3.0.1.jar:/usr/local/src/hadoop-0.20.2/libcommons-logging-1.0.4.jar:/usr/local/src/hadoop-0.20.2/libcommons-logging-api-1.0.4.jar:/usr/local/src/hadoop-0.20.2/libcommons-net-1.4.1.jar:/usr/local/src/hadoop-0.20.2/libcore-3.1.1.jar:/usr/local/src/hadoop-0.20.2/libhsqldb-1.8.0.10.LICENSE.txt:/usr/local/src/hadoop-0.20.2/libhsqldb-1.8.0 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # bunch of stats functions for quick usage in arrays... hobbled together from myself and a bunch of blogs... | |
| module Stats | |
| def sumitup | |
| if self.class == Hash | |
| sum = self.inject(0){|total, item| total = total + (item[1] || 0); total} | |
| end | |
| if self.class == Array | |
| sum = self.inject(0){|total, item| total = total + (item || 0); total} | |
| end |
jamster
/ gist:590624
Created
September 21, 2010 21:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| legacyFields = LegacySchema.NARROW_FIELDS.append( LegacySchemaDOB.NARROW_FIELDS ); | |
| Pipe joinedPipe = new CoGroup( legacyPipe, LegacySchema.PERSON_ID, legacyDOBPipe, LegacyDOBSchema.LEGACY_PERSON_ID, legacyFields, new LeftJoin() ); | |
jamster
/ gist:612369
Created
October 5, 2010 21:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [jasonamster]:~$ curl -s -XGET 'http://localhost:9200/_cluster/state?pretty=true' | |
| { | |
| "cluster_name" : "default", | |
| "master_node" : "539080ef-fc83-49e9-be3a-458a90367386", | |
| "blocks" : { | |
| }, | |
| "nodes" : { | |
| "539080ef-fc83-49e9-be3a-458a90367386" : { | |
| "name" : "Werewolf by Night", | |
| "transport_address" : "inet[/192.168.2.15:9300]", |
jamster
/ gist:613450
Created
October 6, 2010 14:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| curl -s -XGET 'http://localhost:9200/_cluster/state?pretty=true' | |
| { | |
| "cluster_name" : "default", | |
| "master_node" : "539080ef-fc83-49e9-be3a-458a90367386", | |
| "blocks" : { | |
| }, | |
| "nodes" : { | |
| "539080ef-fc83-49e9-be3a-458a90367386" : { | |
| "name" : "Werewolf by Night", | |
| "transport_address" : "inet[/192.168.2.15:9300]", |
jamster
/ gist:634272
Created
October 19, 2010 14:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # I have this data | |
| sample <- c(0, 1, 1, 2, 2, 2, 4, 6, 7, 7, 7, 7, 7, 7, 8, 8, 8, 9, 9, 9, 9) | |
| # I already grouped and counted it | |
| vals <- c(0,1,2,4,6,7,8,9) | |
| freqs <- c(1,2,3,1,1,6,3,4) | |
| # I want to use this, to do a histogram instead of on the original values | |
| # Not hist(sample) | |
| # but hist( vals, freas) |