Skip to content

Instantly share code, notes, and snippets.

View miguno's full-sized avatar

miguno miguno

874 followers · 3 following
View GitHub Profile
@miguno
miguno / Chander Shivdasani
Created March 14, 2012 10:41
Chander Shivdasani
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask$OldOutputCollector.(MapTask.java:573)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
@miguno
miguno / VMWD
Created May 4, 2012 05:30
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask$OldOutputCollector.(MapTask.java:481)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
@miguno
miguno / alberto
Created May 13, 2012 09:00
2012-05-10 23:25:06,585 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = slave/192.168.0.2
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2012-05-10 23:25:12,240 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
2012-05-10 23:25:12,244 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info server at 50010
@miguno
miguno / Sandeep
Created May 30, 2012 13:43
P.S. - Master and slave running same version of Linux and Hadoop and SSH is working perfectly,
because I can start the slave from master node
Also Same settings for core-site.xml, hdfs-site.xml and mapred-site.xml on master(hadooopnode1) and
slave (hadoopnode2)
OS - UBuntu 10
Hadoop Version -
oop@hadoopnode1:~/hadoop-0.20.2/conf$ hadoop version
Hadoop 0.20.2
@miguno
miguno / vignesh
Created June 24, 2012 15:37
chown: changing ownership of `/usr/local/hadoopp/hadoop/libexec/../logs': Operation not permitted
starting namenode, logging to /usr/local/hadoopp/hadoop/libexec/../logs/hadoop-hduser-namenode-ubuntu.out
/usr/local/hadoopp/hadoop/bin/hadoop-daemon.sh: line 135: /usr/local/hadoopp/hadoop/libexec/../logs/hadoop-hduser-namenode-ubuntu.out: Permission denied
head: cannot open `/usr/local/hadoopp/hadoop/libexec/../logs/hadoop-hduser-namenode-ubuntu.out' for reading: No such file or directory
localhost: chown: changing ownership of `/usr/local/hadoopp/hadoop/libexec/../logs': Operation not permitted
localhost: starting datanode, logging to /usr/local/hadoopp/hadoop/libexec/../logs/hadoop-hduser-datanode-ubuntu.out
localhost: /usr/local/hadoopp/hadoop/bin/hadoop-daemon.sh: line 135: /usr/local/hadoopp/hadoop/libexec/../logs/hadoop-hduser-datanode-ubuntu.out: Permission denied
localhost: head: cannot open `/usr/local/hadoopp/hadoop/libexec/../logs/hadoop-hduser-datanode-ubuntu.out' for reading: No such file or direct
@miguno
miguno / gist:3898026
Created October 16, 2012 08:23
hduser@dmnode5:/usr/local/hadoop$ bin/hadoop jar hadoop-examples-1.0.3.jar wordcount -D mapred.reduce.tasks=2 /user/hduser/gutenberg /user/hduser/gutenberg-output
Warning: $HADOOP_HOME is deprecated.
12/10/15 10:34:07 INFO input.FileInputFormat: Total input paths to process : 7
12/10/15 10:34:07 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/10/15 10:34:07 WARN snappy.LoadSnappy: Snappy native library not loaded
12/10/15 10:34:07 INFO mapred.JobClient: Running job: job_201210091154_0010
12/10/15 10:34:08 INFO mapred.JobClient: map 0% reduce 0%
12/10/15 10:34:21 INFO mapred.JobClient: map 28% reduce 0%
12/10/15 10:34:22 INFO mapred.JobClient: map 57% reduce 0%
@miguno
miguno / gist:4549045
Created January 16, 2013 17:36
Hadoop branch-0.20-append build error when running tests
# Build errors for Hadoop branch-0.20-append,
# i.e. SVN revision 1057313 aka git commit df0d79cc aka
# "HDFS-1554. New semantics for [...]" by Hairong Kuang from 10-Jan-2011.
Testsuite: org.apache.hadoop.hdfs.server.namenode.TestEditLogRace
Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
Testcase: testSaveRightBeforeSync took 0 sec
Caused an ERROR
Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.
@miguno
miguno / gist:4549064
Created January 16, 2013 17:37
Another Hadoop branch-0.20-append build error when running tests
# Build errors for Hadoop branch-0.20-append,
# i.e. SVN revision 1057313 aka git commit df0d79cc aka
# "HDFS-1554. New semantics for [...]" by Hairong Kuang from 10-Jan-2011.
Testsuite: org.apache.hadoop.hdfs.TestMultiThreadedSync
Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec
Testcase: testSyncWhileClosing took 0.001 sec
Caused an ERROR
Forked Java VM exited abnormally. Please note the time in the report does not reflect the time until the VM exit.
@miguno
miguno / gist:4605173
Created January 23, 2013 12:48
Re: Hadoop counter visualization

The only open source visualization project that comes to my mind right now is [Twitter Ambrose][1]. You might want to have a look at Ambrose first. It supports the following features in its web UI:

[Ambrose web UI features]

  • A table view of all the associated jobs, along with their current state
  • Chord and graph diagrams to visualize job dependencies and current state
  • An overall script progress bar

Apart from that my personal experience has been with offerings from commercial vendors. Two name but two of them:

@miguno
miguno / KAFKA-0.8_SCALA-2.9.2.md
Last active December 14, 2015 21:19
Building Apache Kafka 0.8 (trunk) for Scala 2.9.2