Shrijeet shrijeet
shrijeet
/ StringFunnel.java
Created
October 9, 2013 07:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.example; | |
| import com.google.common.hash.Funnel; | |
| import com.google.common.hash.PrimitiveSink; | |
| public enum StringFunnel implements Funnel<String> { | |
| INSTANCE; | |
| @Override | |
| public void funnel(String from, PrimitiveSink into) { |
shrijeet
/ TestMRCJCFileOutputCommitter_test_fail.java
Created
November 4, 2014 21:45
MAPREDUCE-4815 fails test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ------------------------------------------------------- | |
| T E S T S | |
| ------------------------------------------------------- | |
| Running org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter | |
| Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 1.612 sec <<< FAILURE! - in org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter | |
| testAbort(org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter) Time elapsed: 0.116 sec <<< ERROR! | |
| java.lang.NullPointerException: null | |
| at org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter.testAbort(TestMRCJCFileOutputCommitter.java:144) | |
| testFailAbort(org.apache.hadoop.mapred.TestMRCJCFileOutputCommitter) Time elapsed: 0.071 sec <<< FAILURE! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [shrijeet@xxx-209 ~]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 5 5 | |
| Number of Maps = 5 | |
| Samples per Map = 5 | |
| Wrote input for Map #0 | |
| Wrote input for Map #1 | |
| Wrote input for Map #2 | |
| Wrote input for Map #3 | |
| Wrote input for Map #4 | |
| Starting Job | |
| 14/11/04 16:53:51 INFO client.RMProxy: Connecting to ResourceManager at yarn-rm.xxx-yyyy.example.net/172.0.0.0:8032 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @Override | |
| public void cleanup(Context context) throws IOException { | |
| //write output to a file | |
| Configuration conf = context.getConfiguration(); | |
| Path outDir = new Path(conf.get(FileOutputFormat.OUTDIR)); | |
| Path outFile = new Path(outDir, "reduce-out"); | |
| FileSystem fileSys = FileSystem.get(conf); | |
| SequenceFile.Writer writer = SequenceFile.createWriter(fileSys, conf, | |
| outFile, LongWritable.class, LongWritable.class, | |
| CompressionType.NONE); |
shrijeet
/ AppSchedulable.java
Created
March 23, 2015 16:27
Only one of local, rack or any is satisfied
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| if (rackLocalRequest != null && rackLocalRequest.getNumContainers() != 0 | |
| && localRequest != null && localRequest.getNumContainers() != 0) { | |
| return assignContainer(node, localRequest, | |
| NodeType.NODE_LOCAL, reserved); | |
| } | |
| if (rackLocalRequest != null && !rackLocalRequest.getRelaxLocality()) { | |
| continue; | |
| } |
shrijeet
/ quickstart-hdfs.properties
Created
July 9, 2016 19:34
kafka-connect-hdfs quickstart-hdfs.properties
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name=hdfs-sink | |
| connector.class=io.confluent.connect.hdfs.HdfsSinkConnector | |
| tasks.max=1 | |
| topics=test_hdfs | |
| hdfs.url=hdfs://10.211.55.101:8020 | |
| flush.size=3 | |
| hive.integration=true | |
| hive.metastore.uris=thrift://localhost:9083 | |
| schema.compatibility=BACKWARD | |
| format.class=io.confluent.connect.hdfs.parquet.ParquetFormat |
OlderNewer