This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This gist demonstrates how to do a map-side join, loading one small dataset from DistributedCache into a HashMap | |
in memory, and joining with a larger dataset. | |
Includes: | |
--------- | |
1. Input data and script download | |
2. Dataset structure review | |
3. Expected results | |
4. Mapper code | |
5. Driver code |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
About this gist: | |
================ | |
This gist is a part of a series of log parsers in Java Mapreduce, Pig, Hive, Python... | |
This one covers a log parser in Cascading. | |
It reads syslogs in HDFS - | |
a) Parses them based on a regex pattern & writes parsed files to HDFS | |
b) Writes records that dont match pattern to HDFS | |
c) Writes a report to HDFS that contains the count of distinct processes logged. | |
Other gists/blogs: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
...... | |
List<String> artifactList = new List<String> (); | |
var scanOpts = new ScanOptions(); | |
String rowRegex = rowID + ".*"; | |
IteratorSetting iterSttng = new IteratorSetting(); | |
iterSttng.Priority = 15; | |
iterSttng.Name = "rowIDRegexFilter"; | |
iterSttng.IteratorClass="org.apache.accumulo.core.iterators.user.RegExFilter"; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This gist covers a simple Hive genericUDF in Java, that mimics NVL2 functionality in Oracle. | |
NVL2 is used to handle nulls and conditionally substitute values. | |
Included: | |
1. Input data | |
2. Expected results | |
3. UDF code in java | |
4. Hive query to demo the UDF | |
5. Output | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This gist covers a simple Hive eval UDF in Java, that mimics NVL2 functionality in Oracle. | |
NVL2 is used to handle nulls and conditionally substitute values. | |
Included: | |
1. Input data | |
2. Expected results | |
3. UDF code in java | |
4. Hive query to demo the UDF | |
5. Output |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This gist covers a simple Pig eval UDF in Java, that mimics NVL2 functionality in Oracle. | |
Included: | |
1. Input data | |
2. UDF code in java | |
3. Pig script to demo the UDF | |
4. Expected result | |
5. Command to execute script | |
6. Output |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
********************** | |
**Gist | |
********************** | |
This gist details how to inner join two large datasets on the map-side, leveraging the join capability | |
in mapreduce. Such a join makes sense if both input datasets are too large to qualify for distribution | |
through distributedcache, and can be implemented if both input datasets can be joined by the join key | |
and both input datasets are sorted in the same order, by the join key. | |
There are two critical pieces to engaging the join behavior: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
************************* | |
Gist | |
************************* | |
One more gist related to controlling the number of mappers in a mapreduce task. | |
Background on Inputsplits | |
-------------------------- | |
An inputsplit is a chunk of the input data allocated to a map task for processing. FileInputFormat | |
generates inputsplits (and divides the same into records) - one inputsplit for each file, unless the |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Secondary sort in Mapreduce | |
With mapreduce framework, the keys are sorted but the values associated with each key | |
are not. In order for the values to be sorted, we need to write code to perform what is | |
referred to a secondary sort. The sample code in this gist demonstrates such a sort. | |
The input to the program is a bunch of employee attributes. | |
The output required is department number (deptNo) in ascending order, and the employee last name, | |
first name and employee ID in descending order. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This gist demonstrates how to do a map-side join, joining a MapFile from distributedcache | |
with a larger dataset in HDFS. | |
Includes: | |
--------- | |
1. Input data and script download | |
2. Dataset structure review | |
3. Expected results | |
4. Mapper code | |
5. Driver code |
OlderNewer