Esme Mora alienrobotwizard
alienrobotwizard
/ gist:702077
Created
November 16, 2010 17:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| java.lang.RuntimeException: java.io.IOException: Invalid localDeleteTime read: 0 | |
| at org.apache.cassandra.db.BinaryVerbHandler.doVerb(BinaryVerbHandler.java:54) | |
| at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:50) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) | |
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) | |
| at java.lang.Thread.run(Thread.java:619) | |
| Caused by: java.io.IOException: Invalid localDeleteTime read: 0 | |
| at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:346) | |
| at org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:298) | |
| at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:125) |
alienrobotwizard
/ ElasticBulkLoader.java
Created
October 22, 2010 20:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.io.IOException; | |
| import java.io.UnsupportedEncodingException; | |
| import java.math.BigInteger; | |
| import java.net.URI; | |
| import java.net.URISyntaxException; | |
| import java.util.ArrayList; | |
| import java.util.Iterator; | |
| import java.util.LinkedList; | |
| import java.util.List; |
alienrobotwizard
/ gist:641287
Created
October 22, 2010 20:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 2010-10-22 17:58:56,590 WARN transport: [Poundcakes] Transport response handler timed out, action [discovery/zen/fd/masterPing], node [[Spider-Slayer][62863483-6ad6-4531-b3bf-75a34ecd0e96][inet[/10.204.153.212:9300]]] | |
| 2010-10-22 18:00:32,544 WARN transport: [Poundcakes] Transport response handler timed out, action [discovery/zen/fd/masterPing], node [[Spider-Slayer][62863483-6ad6-4531-b3bf-75a34ecd0e96][inet[/10.204.153.212:9300]]] | |
| 2010-10-22 18:00:32,544 WARN transport: [Poundcakes] Transport response handler timed out, action [discovery/zen/fd/masterPing], node [[Spider-Slayer][62863483-6ad6-4531-b3bf-75a34ecd0e96][inet[/10.204.153.212:9300]]] |
alienrobotwizard
/ cassandra_via_hector_with_jruby.rb
Created
October 4, 2010 02:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env ruby | |
| require 'java' | |
| hector_lib = File.join(ENV['HOME'], "Programming/hector/releases/hector-0.7.0-18") | |
| hector_target = File.join(ENV['HOME'], "Programming/hector/target") | |
| Dir["#{hector_target}/*.jar", "#{hector_lib}/*.jar"].each{|jar| $CLASSPATH << jar} | |
| java_import 'me.prettyprint.cassandra.serializers.StringSerializer' | |
| java_import 'me.prettyprint.hector.api.Cluster' | |
| java_import 'me.prettyprint.hector.api.Keyspace' |
alienrobotwizard
/ gml_parser.rb
Created
October 2, 2010 20:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| module GMLParser | |
| class Graph < Struct.new( | |
| :nodes, | |
| :edges | |
| ) | |
| def edge_list | |
| self.edges.map{|e| [e[:source], e[:target], e[:value]]} | |
| end |
alienrobotwizard
/ avromapper.rb
Created
August 25, 2010 03:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| require 'rubygems' | |
| require 'avro' | |
| require 'wukong' | |
| Time.class_eval do | |
| def self.epoch_microseconds | |
| (Time.now.to_i * 1_000_000) | |
| end | |
| end |
alienrobotwizard
/ some_twitter_folks.tsv
Created
August 2, 2010 22:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| bizmarketing4u | 49108829 | 0.094361630 | 1 | |
|---|---|---|---|---|
| BrowneBig570 | 50190727 | 0.144720420 | 1 | |
| Megan___Fox | 49509322 | 0.147560500 | 1 | |
| pen2netone | 47910899 | 0.064650595 | 1 | |
| dextradyoung | 47502802 | 0.146222870 | 2 | |
| mbainstitute | 41130608 | 0.864213050 | 2 | |
| mcspartan76 | 17034645 | 0.111582670 | 2 | |
| Opereur2u | 65572992 | 0.138023360 | 2 | |
| shaunaconway3 | 63460580 | 0.105175970 | 2 | |
| BrianaPitts | 69789022 | 0.163385990 | 3 |
alienrobotwizard
/ parse.sh
Created
July 29, 2010 19:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| cat 200_twitspam_2.json| ruby -ne 'puts ["twitter_user_timeline_request", 15491144, 3, "0", "http://twitter.com/statuses/user_timeline/15491144.json?&page=1&count=200", 20100729, 200, "foobar", $_.strip].join("\t")' | ~/Programming/infochimps-data/social/network/twitter/base/parse/parse_twitter_api_requests.rb --map > 200_twitspam_parsed_2.tsv |
alienrobotwizard
/ example.json
Created
July 28, 2010 02:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 4.2.145 [[0,{"latitude":"38.0000","country_code":"US","longitude":"-97.0000"}],[159,{"latitude":"38.0000","country_code":"US","longitude":"-97.0000"}],[160,{"household_income":"41506","percent_hispanic":"17.98","city":"Irving","percent_semi_permanent":"18.61","percent_asian":"14.45","percent_under_18":"20.06","latitude":"32.8791","per_capita_income":"28214","percent_bs_graduate":"16.34","area_code":"623","country_code":"US","zip_code":"75038","percent_below_poverty":"10.61","people_per_household":"2.0","housing_unit_value":"149500","percent_homeownership":"11.27","housing_units":"13305","work_travel_time":"3.1","percent_dual_race":"3.09","percent_pacific":"0.0","percent_white":"50.03","population":"25191","region_code":"TX","percent_hs_graduate":"16.34","percent_non_english":"36.09","percent_foreign":"28.9","percent_black":"23.75","percent_over_65":"1.87","metro_code":"972","longitude":"-96.9898","households":"12466","percent_under_5":"7.3","percent_native":"0.56","percent_female":"48.11"}],[191,{"household_i |
alienrobotwizard
/ example.job
Created
July 27, 2010 21:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # Example azkaban job. Assumes you have two MR jobs to be run sequentially. | |
| # | |
| type=command | |
| command=$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-*streaming*.jar -input /path/to/data -output /path/to/outputA -mapper mapperA.py -reducer reducerA.py | |
| command.1=$HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-*streaming*.jar -input /path/to/outputA -output /path/to/outputB -mapper mapperB.py -reducer reducerB.py |