Sachin Thirumala stdatalabs
stdatalabs
/ gist:cb57f4d7cfd8ac5bf89476d28bb64e6e
Created
December 24, 2017 15:35
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 0x8e8CF9467c121897F86BccB0C6Cfb165AAA7482A |
stdatalabs
/ customRecordReader.java
Last active
October 5, 2017 19:44
A customRecordReader class to read every input split. More @ stdatalabs.blogspot.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.io.IOException; | |
| import java.io.InputStream; | |
| import org.apache.commons.logging.Log; | |
| import org.apache.commons.logging.LogFactory; | |
| import org.apache.hadoop.conf.Configuration; | |
| import org.apache.hadoop.fs.FSDataInputStream; | |
| import org.apache.hadoop.fs.FileSystem; | |
| import org.apache.hadoop.fs.Path; | |
| import org.apache.hadoop.io.LongWritable; |
stdatalabs
/ CustomTextInputFormat.java
Last active
October 24, 2016 08:03
A CustomTextInputFormat class that extends from TextInputFormat and calls the customRecordReader class. More @ stdatalabs.blogspot.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.io.IOException; | |
| import org.apache.hadoop.io.LongWritable; | |
| import org.apache.hadoop.io.Text; | |
| import org.apache.hadoop.mapred.FileSplit; | |
| import org.apache.hadoop.mapred.InputSplit; | |
| import org.apache.hadoop.mapred.JobConf; | |
| import org.apache.hadoop.mapred.RecordReader; | |
| import org.apache.hadoop.mapred.Reporter; | |
| import org.apache.hadoop.mapred.TextInputFormat; |
stdatalabs
/ Hive customInputFormat - pom.xml
Created
October 9, 2016 07:57
Dependencies for hive customInputFormat
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <dependency> | |
| <groupId>org.apache.hive</groupId> | |
| <artifactId>hive-exec</artifactId> | |
| <version>0.11.0</version> | |
| </dependency> | |
| <dependency> | |
| <groupId>org.apache.hadoop</groupId> | |
| <artifactId>hadoop-common</artifactId> | |
| <version>2.7.1</version> | |
| </dependency> |
stdatalabs
/ JsonWordSplitterBolt.java
Last active
October 24, 2016 08:02
A storm bolt to split tuples sent by KafkaSpout. More @ stdatalabs.blogspot.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import backtype.storm.task.OutputCollector; | |
| import backtype.storm.task.TopologyContext; | |
| import backtype.storm.topology.OutputFieldsDeclarer; | |
| import backtype.storm.topology.base.BaseRichBolt; | |
| import backtype.storm.tuple.Fields; | |
| import backtype.storm.tuple.Tuple; | |
| import backtype.storm.tuple.Values; | |
| import twitter4j.Status; | |
| import java.util.Map; |
stdatalabs
/ TwitterSampleSpout.java
Last active
September 29, 2021 14:50
A storm spout to produce a stream of tuples from the twitter streaming API. More @ stdatalabs.blogspot.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.util.Map; | |
| import java.util.concurrent.LinkedBlockingQueue; | |
| import twitter4j.FilterQuery; | |
| import twitter4j.StallWarning; | |
| import twitter4j.Status; | |
| import twitter4j.StatusDeletionNotice; | |
| import twitter4j.StatusListener; | |
| import twitter4j.TwitterStream; |
stdatalabs
/ TwitterWordCountTopology.java
Last active
October 24, 2016 08:00
A storm topology to count the list of top words used in tweets regarding a topic. More @ stdatalabs.blogspot.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.util.*; | |
| import com.stdatalabs.Storm.IgnoreWordsBolt; | |
| import com.stdatalabs.Storm.TwitterSampleSpout; | |
| import com.stdatalabs.Storm.WordCounterBolt; | |
| import com.stdatalabs.Storm.JsonWordSplitterBolt; | |
| import backtype.storm.tuple.Fields; | |
| import backtype.storm.tuple.Values; | |
| import backtype.storm.Config; |
stdatalabs
/ dbms_crypto_UDF.scala
Last active
January 23, 2019 10:30
A Spark UDF to find the MD5 message digest of a column. More @ stdatalabs.blogspot.com
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import org.apache.spark.{SparkConf, SparkContext} | |
| import org.apache.spark.sql._ | |
| import org.apache.spark.sql.hive.HiveContext | |
| val hiveContext = new HiveContext(sc) | |
| import hiveContext.implicits._ | |
| import hiveContext.sql | |
| import sqlContext.implicits._ | |
| import java.security.MessageDigest | |
| /** |
stdatalabs
/ KafkaSparkPopularHashTags.scala
Last active
November 11, 2017 03:26
A Spark Streaming - Kafka integration to receive twitter data from kafka topic and find the popular hashtags. More @ stdatalabs.blogspot.com
stdatalabs
/ Kafka - pom.xml
Created
October 4, 2016 17:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka_2.10 --> | |
| <dependency> | |
| <groupId>org.apache.kafka</groupId> | |
| <artifactId>kafka_2.10</artifactId> | |
| <version>0.8.2.1</version> | |
| </dependency> | |
| <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-core --> | |
| <dependency> | |
| <groupId>org.twitter4j</groupId> | |
| <artifactId>twitter4j-core</artifactId> |
NewerOlder