Skip to content

Instantly share code, notes, and snippets.

@aperiodic
Created November 1, 2011 19:48
Show Gist options
  • Save aperiodic/1331700 to your computer and use it in GitHub Desktop.
Save aperiodic/1331700 to your computer and use it in GitHub Desktop.
Failed reduce task's logs
2011-11-01 11:12:59,373 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.system.dir; Ignoring.
2011-11-01 11:12:59,374 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.local.dir; Ignoring.
2011-11-01 11:12:59,374 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.tasktracker.reduce.tasks.maximum; Ignoring.
2011-11-01 11:12:59,415 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.tasktracker.map.tasks.maximum; Ignoring.
2011-11-01 11:12:59,416 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.child.java.opts; Ignoring.
2011-11-01 11:12:59,417 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.reduce.parallel.copies; Ignoring.
2011-11-01 11:12:59,418 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.child.ulimit; Ignoring.
2011-11-01 11:12:59,419 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.jobtracker.taskScheduler; Ignoring.
2011-11-01 11:12:59,420 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: fs.default.name; Ignoring.
2011-11-01 11:12:59,420 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.reduce.max.attempts; Ignoring.
2011-11-01 11:12:59,420 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.job.tracker; Ignoring.
2011-11-01 11:12:59,480 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2011-11-01 11:12:59,835 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=SHUFFLE, sessionId=
2011-11-01 11:13:00,174 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.system.dir; Ignoring.
2011-11-01 11:13:00,174 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.local.dir; Ignoring.
2011-11-01 11:13:00,174 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.tasktracker.reduce.tasks.maximum; Ignoring.
2011-11-01 11:13:00,174 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: dfs.data.dir; Ignoring.
2011-11-01 11:13:00,175 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: dfs.support.append; Ignoring.
2011-11-01 11:13:00,176 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.tasktracker.map.tasks.maximum; Ignoring.
2011-11-01 11:13:00,176 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.child.java.opts; Ignoring.
2011-11-01 11:13:00,176 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: dfs.namenode.handler.count; Ignoring.
2011-11-01 11:13:00,177 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.reduce.parallel.copies; Ignoring.
2011-11-01 11:13:00,177 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.child.ulimit; Ignoring.
2011-11-01 11:13:00,177 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.jobtracker.taskScheduler; Ignoring.
2011-11-01 11:13:00,178 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: dfs.datanode.max.xcievers; Ignoring.
2011-11-01 11:13:00,178 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: fs.default.name; Ignoring.
2011-11-01 11:13:00,178 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.reduce.max.attempts; Ignoring.
2011-11-01 11:13:00,178 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: mapred.job.tracker; Ignoring.
2011-11-01 11:13:00,178 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: dfs.name.dir; Ignoring.
2011-11-01 11:13:00,178 WARN org.apache.hadoop.conf.Configuration: /mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/job.xml:a attempt to override final parameter: dfs.datanode.handler.count; Ignoring.
2011-11-01 11:13:00,355 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-cdh3u0--1, built on 03/26/2011 00:24 GMT
2011-11-01 11:13:00,355 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=ip-10-204-102-239.ec2.internal
2011-11-01 11:13:00,355 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.6.0_26
2011-11-01 11:13:00,355 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
2011-11-01 11:13:00,355 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.26/jre
2011-11-01 11:13:00,356 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/usr/lib/hadoop-0.20/conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-core-0.20.2-cdh3u0.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u0.jar:/usr/lib/hadoop-0.20/lib/hadoop-lzo-0.4.10.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hbase/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/usr/lib/hbase/bin/..:/usr/lib/hbase/bin/../hbase-0.90.1-cdh3u0.jar:/usr/lib/hbase/bin/../hbase-0.90.1-cdh3u0-tests.jar:/usr/lib/hbase/bin/../lib/activation-1.1.jar:/usr/lib/hbase/bin/../lib/asm-3.1.jar:/usr/lib/hbase/bin/../lib/avro-1.3.3.jar:/usr/lib/hbase/bin/../lib/commons-cli-1.2.jar:/usr/lib/hbase/bin/../lib/commons-codec-1.4.jar:/usr/lib/hbase/bin/../lib/commons-el-1.0.jar:/usr/lib/hbase/bin/../lib/commons-httpclient-3.1.jar:/usr/lib/hbase/bin/../lib/commons-lang-2.5.jar:/usr/lib/hbase/bin/../lib/commons-logging-1.1.1.jar:/usr/lib/hbase/bin/../lib/commons-net-1.4.1.jar:/usr/lib/hbase/bin/../lib/core-3.1.1.jar:/usr/lib/hbase/bin/../lib/guava-r06.jar:/usr/lib/hbase/bin/../lib/hadoop-core.jar:/usr/lib/hbase/bin/../lib/hadoop-lzo-0.4.10.jar:/usr/lib/hbase/bin/../lib/hbase-0.90.1-cdh3u0.jar:/usr/lib/hbase/bin/../lib/jackson-core-asl-1.5.2.jar:/usr/lib/hbase/bin/../lib/jackson-jaxrs-1.5.5.jar:/usr/lib/hbase/bin/../lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hbase/bin/../lib/jackson-xc-1.5.5.jar:/usr/lib/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/usr/lib/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/usr/lib/hbase/bin/../lib/jaxb-api-2.1.jar:/usr/lib/hbase/bin/../lib/jaxb-impl-2.1.12.jar:/usr/lib/hbase/bin/../lib/jersey-core-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-json-1.4.jar:/usr/lib/hbase/bin/../lib/jersey-server-1.4.jar:/usr/lib/hbase/bin/../lib/jettison-1.1.jar:/usr/lib/hbase/bin/../lib/jetty-6.1.26.jar:/usr/lib/hbase/bin/../lib/jetty-util-6.1.26.jar:/usr/lib/hbase/bin/../lib/jruby-complete-1.0.3.jar:/usr/lib/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/lib/hbase/bin/../lib/jsp-api-2.1.jar:/usr/lib/hbase/bin/../lib/jsr311-api-1.1.1.jar:/usr/lib/hbase/bin/../lib/log4j-1.2.16.jar:/usr/lib/hbase/bin/../lib/protobuf-java-2.3.0.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hbase/bin/../lib/servlet-api-2.5.jar:/usr/lib/hbase/bin/../lib/slf4j-api-1.5.8.jar:/usr/lib/hbase/bin/../lib/slf4j-log4j12-1.5.8.jar:/usr/lib/hbase/bin/../lib/stax-api-1.0.1.jar:/usr/lib/hbase/bin/../lib/thrift-0.2.0.jar:/usr/lib/hbase/bin/../lib/xmlenc-0.52.jar:/usr/lib/hbase/bin/../lib/zookeeper.jar:/etc/zookeeper:/etc/hadoop-0.20/conf:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u0-test.jar:/usr/lib/hadoop-0.20/hadoop-tools.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u0-tools.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u0-ant.jar:/usr/lib/hadoop-0.20/hadoop-examples.jar:/usr/lib/hadoop-0.20/hadoop-test-0.20.2-cdh3u0.jar:/usr/lib/hadoop-0.20/hadoop-ant.jar:/usr/lib/hadoop-0.20/hadoop-core-0.20.2-cdh3u0.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u0-examples.jar:/usr/lib/hadoop-0.20/hadoop-test.jar:/usr/lib/hadoop-0.20/hadoop-ant-0.20.2-cdh3u0.jar:/usr/lib/hadoop-0.20/hadoop-tools-0.20.2-cdh3u0.jar:/usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u0-core.jar:/usr/lib/hadoop-0.20/hadoop-core.jar:/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u0.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.26.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/aspectjrt-1.6.5.jar:/usr/lib/hadoop-0.20/lib/jackson-core-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/aspectjtools-1.6.5.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/jetty-servlet-tester-6.1.26.jar:/usr/lib/hadoop-0.20/lib/jackson-mapper-asl-1.5.2.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/commons-daemon-1.0.1.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/hadoop-fairscheduler-0.20.2-cdh3u0.jar:/usr/lib/hadoop-0.20/lib/mockito-all-1.8.2.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-20081211.jar:/usr/lib/hadoop-0.20/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/hadoop-lzo-0.4.10.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20/lib/junit-4.5.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.4.jar:/usr/lib/zookeeper/zookeeper-3.3.3-cdh3u0.jar:/usr/lib/zookeeper/zookeeper.jar:/usr/lib/zookeeper/lib/log4j-1.2.15.jar:/usr/lib/zookeeper/lib/jline-0.9.94.jar:/mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/jars/classes:/mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/jars/job.jar:/mnt/data/hadoop/mapred/local/taskTracker/ubuntu/distcache/-277074928334866505_-808452054_1583959804/ip-10-118-65-68.ec2.internal/tmp/hadoop-mapred/mapred/staging/ubuntu/.staging/job_201110251936_0044/libjars/hadoop-core-0.20.2-cdh3u1.jar:/mnt/data/hadoop/mapred/local/taskTracker/ubuntu/distcache/6575547229097817614_-370382403_1583960662/ip-10-118-65-68.ec2.internal/tmp/hadoop-mapred/mapred/staging/ubuntu/.staging/job_201110251936_0044/libjars/mapreduce.jar:/mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/attempt_201110251936_0044_r_000005_0/work
2011-11-01 11:13:00,356 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64:/mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/attempt_201110251936_0044_r_000005_0/work
2011-11-01 11:13:00,356 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/attempt_201110251936_0044_r_000005_0/work/tmp
2011-11-01 11:13:00,356 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2011-11-01 11:13:00,356 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux
2011-11-01 11:13:00,356 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64
2011-11-01 11:13:00,356 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.35-24-virtual
2011-11-01 11:13:00,356 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=mapred
2011-11-01 11:13:00,356 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/usr/lib/hadoop-0.20
2011-11-01 11:13:00,356 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/mnt/data/hadoop/mapred/local/taskTracker/ubuntu/jobcache/job_201110251936_0044/attempt_201110251936_0044_r_000005_0/work
2011-11-01 11:13:00,358 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-118-65-68.ec2.internal:2181 sessionTimeout=180000 watcher=hconnection
2011-11-01 11:13:00,371 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ip-10-118-65-68.ec2.internal/10.118.65.68:2181
2011-11-01 11:13:00,373 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ip-10-118-65-68.ec2.internal/10.118.65.68:2181, initiating session
2011-11-01 11:13:00,381 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server ip-10-118-65-68.ec2.internal/10.118.65.68:2181, sessionid = 0x131b1dbb3742a40, negotiated timeout = 60000
2011-11-01 11:13:00,562 INFO org.apache.hadoop.hbase.mapreduce.TableOutputFormat: Created table instance for twitter_sample.2-wordcounts-daily
2011-11-01 11:13:00,601 INFO org.apache.hadoop.mapred.ReduceTask: ShuffleRamManager: MemoryLimit=668126400, MaxSingleShuffleLimit=167031600
2011-11-01 11:13:00,607 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Thread started: Thread for merging on-disk files
2011-11-01 11:13:00,607 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Thread waiting: Thread for merging on-disk files
2011-11-01 11:13:00,609 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Need another 17 map output(s) where 0 is already in progress
2011-11-01 11:13:00,609 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Thread started: Thread for polling Map Completion Events
2011-11-01 11:13:00,609 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 0 outputs (0 slow hosts and0 dup hosts)
2011-11-01 11:13:00,610 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Thread started: Thread for merging in memory files
2011-11-01 11:13:05,610 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 2 outputs (0 slow hosts and0 dup hosts)
2011-11-01 11:13:06,108 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and8 dup hosts)
2011-11-01 11:13:06,114 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and8 dup hosts)
2011-11-01 11:13:06,118 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and8 dup hosts)
2011-11-01 11:13:06,120 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and8 dup hosts)
2011-11-01 11:13:06,121 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and8 dup hosts)
2011-11-01 11:13:11,151 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and2 dup hosts)
2011-11-01 11:13:11,155 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and2 dup hosts)
2011-11-01 11:13:11,156 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and2 dup hosts)
2011-11-01 11:13:11,159 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and2 dup hosts)
2011-11-01 11:13:11,160 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and2 dup hosts)
2011-11-01 11:13:11,202 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and2 dup hosts)
2011-11-01 11:13:11,249 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and2 dup hosts)
2011-11-01 11:13:11,251 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and2 dup hosts)
2011-11-01 11:13:12,690 INFO org.apache.hadoop.mapred.ReduceTask: Read 210752974 bytes from map-output for attempt_201110251936_0044_m_000008_0
2011-11-01 11:13:12,691 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Thread waiting: Thread for merging on-disk files
2011-11-01 11:13:12,698 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2011-11-01 11:13:12,698 INFO org.apache.hadoop.mapred.ReduceTask: attempt_201110251936_0044_r_000005_0 Scheduled 1 outputs (0 slow hosts and0 dup hosts)
2011-11-01 11:13:13,648 INFO org.apache.hadoop.mapred.ReduceTask: GetMapEventsThread exiting
2011-11-01 11:13:13,649 INFO org.apache.hadoop.mapred.ReduceTask: getMapsEventsThread joined.
2011-11-01 11:13:13,649 INFO org.apache.hadoop.mapred.ReduceTask: Closed ram manager
2011-11-01 11:13:13,650 INFO org.apache.hadoop.mapred.ReduceTask: Interleaved on-disk merge complete: 1 files left.
2011-11-01 11:13:13,650 INFO org.apache.hadoop.mapred.ReduceTask: In-memory merge complete: 16 files left.
2011-11-01 11:13:13,657 INFO org.apache.hadoop.mapred.Merger: Merging 16 sorted segments
2011-11-01 11:13:13,657 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 128208007 bytes
2011-11-01 11:13:14,575 INFO org.apache.hadoop.mapred.ReduceTask: Merged 16 segments, 128208037 bytes to disk to satisfy reduce memory limit
2011-11-01 11:13:14,576 INFO org.apache.hadoop.mapred.ReduceTask: Merging 2 files, 338960985 bytes from disk
2011-11-01 11:13:14,577 INFO org.apache.hadoop.mapred.ReduceTask: Merging 0 segments, 0 bytes from memory into reduce
2011-11-01 11:13:14,577 INFO org.apache.hadoop.mapred.Merger: Merging 2 sorted segments
2011-11-01 11:13:14,580 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 338960977 bytes
2011-11-01 11:13:17,077 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=ip-10-118-65-68.ec2.internal:2181 sessionTimeout=180000 watcher=hconnection
2011-11-01 11:13:17,110 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server ip-10-118-65-68.ec2.internal/10.118.65.68:2181
2011-11-01 11:13:17,111 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to ip-10-118-65-68.ec2.internal/10.118.65.68:2181, initiating session
2011-11-01 11:13:17,115 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server ip-10-118-65-68.ec2.internal/10.118.65.68:2181, sessionid = 0x131b1dbb3742a41, negotiated timeout = 60000
2011-11-01 11:12:58,156 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201110251936_0044_r_000005_0 task's state:UNASSIGNED
2011-11-01 11:12:58,156 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201110251936_0044_r_000005_0 which needs 1 slots
2011-11-01 11:12:58,156 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201110251936_0044_r_000005_0 which needs 1 slots
2011-11-01 11:12:58,167 INFO org.apache.hadoop.mapred.JvmManager: Killing JVM: jvm_201110251936_0044_r_992760938
2011-11-01 11:12:58,241 INFO org.apache.hadoop.util.ProcessTree: Killing process group22479 with signal TERM. Exit code 0
2011-11-01 11:12:58,241 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201110251936_0044_r_-1341282920
2011-11-01 11:12:58,241 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201110251936_0044_r_-1341282920 spawned.
2011-11-01 11:12:58,243 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /mnt/data/hadoop/mapred/local/ttprivate/taskTracker/ubuntu/jobcache/job_201110251936_0044/attempt_201110251936_0044_r_000005_0/taskjvm.sh
2011-11-01 11:12:58,728 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201110251936_0044_r_992760938 exited with exit code 0. Number of tasks it ran: 1
2011-11-01 11:12:59,206 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201110251936_0044_r_-1341282920 given task: attempt_201110251936_0044_r_000005_0
2011-11-01 11:13:06,081 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.204.102.239:50060, dest: 10.204.102.239:49939, bytes: 6, op: MAPRED_SHUFFLE, cliID: attempt_201110251936_0044_m_000005_0, duration: 294248
2011-11-01 11:13:06,110 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.204.102.239:50060, dest: 10.204.102.239:49939, bytes: 6, op: MAPRED_SHUFFLE, cliID: attempt_201110251936_0044_m_000010_0, duration: 280789
2011-11-01 11:13:06,115 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.204.102.239:50060, dest: 10.204.102.239:49939, bytes: 6, op: MAPRED_SHUFFLE, cliID: attempt_201110251936_0044_m_000001_0, duration: 213737
2011-11-01 11:13:06,119 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.204.102.239:50060, dest: 10.204.102.239:49939, bytes: 6, op: MAPRED_SHUFFLE, cliID: attempt_201110251936_0044_m_000003_0, duration: 188124
2011-11-01 11:13:06,120 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.204.102.239:50060, dest: 10.204.102.239:49939, bytes: 6, op: MAPRED_SHUFFLE, cliID: attempt_201110251936_0044_m_000007_0, duration: 175444
2011-11-01 11:13:06,461 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.098039225% reduce > copy (5 of 17 at 0.00 MB/s) >
2011-11-01 11:13:09,491 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.098039225% reduce > copy (5 of 17 at 0.00 MB/s) >
2011-11-01 11:13:12,521 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.27450982% reduce > copy (14 of 17 at 11.12 MB/s) >
2011-11-01 11:13:12,689 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.204.102.239:50060, dest: 10.204.102.239:49939, bytes: 210752974, op: MAPRED_SHUFFLE, cliID: attempt_201110251936_0044_m_000008_0, duration: 6567552460
2011-11-01 11:13:12,695 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.204.102.239:50060, dest: 10.204.102.239:49939, bytes: 6, op: MAPRED_SHUFFLE, cliID: attempt_201110251936_0044_m_000012_0, duration: 450675
2011-11-01 11:13:12,700 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 10.204.102.239:50060, dest: 10.204.102.239:49939, bytes: 6, op: MAPRED_SHUFFLE, cliID: attempt_201110251936_0044_m_000014_0, duration: 220748
2011-11-01 11:13:13,652 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.27450982% reduce > copy (14 of 17 at 11.12 MB/s) >
2011-11-01 11:13:14,581 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.27450982% reduce > copy (14 of 17 at 11.12 MB/s) >
2011-11-01 11:13:15,557 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.6666667% reduce > reduce
2011-11-01 11:13:18,558 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.6667873% processing row indexed 56 > reduce
2011-11-01 11:13:21,564 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.6701967% processing row indexed 1875 > reduce
2011-11-01 11:13:24,579 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.67512697% processing row indexed 4619 > reduce
2011-11-01 11:13:27,581 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.6801064% processing row indexed 7309 > reduce
2011-11-01 11:13:30,583 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.68501174% processing row indexed 10042 > reduce
2011-11-01 11:13:33,585 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.68950695% processing row indexed 12457 > reduce
2011-11-01 11:13:36,587 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.69469225% processing row indexed 15344 > reduce
2011-11-01 11:13:39,588 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.6993966% processing row indexed 17880 > reduce
2011-11-01 11:13:42,590 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.70487213% processing row indexed 20903 > reduce
2011-11-01 11:13:45,592 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.71010435% processing row indexed 23761 > reduce
2011-11-01 11:13:48,616 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7148492% processing row indexed 26341 > reduce
2011-11-01 11:13:51,623 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7193462% processing row indexed 28828 > reduce
2011-11-01 11:13:54,212 INFO org.mortbay.log: org.mortbay.io.nio.SelectorManager$SelectSet@4f163cdc JVM BUG(s) - injecting delay6 times
2011-11-01 11:13:54,212 INFO org.mortbay.log: org.mortbay.io.nio.SelectorManager$SelectSet@4f163cdc JVM BUG(s) - recreating selector 6 times, canceled keys 102 times
2011-11-01 11:13:54,625 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7247996% processing row indexed 31783 > reduce
2011-11-01 11:13:57,627 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.73003495% processing row indexed 34702 > reduce
2011-11-01 11:14:00,629 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.73519105% processing row indexed 37488 > reduce
2011-11-01 11:14:03,633 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.73823357% processing row indexed 39164 > reduce
2011-11-01 11:14:06,635 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.74348474% processing row indexed 42067 > reduce
2011-11-01 11:14:09,636 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7485377% processing row indexed 44835 > reduce
2011-11-01 11:14:12,638 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7538707% processing row indexed 47744 > reduce
2011-11-01 11:14:15,640 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7589345% processing row indexed 50496 > reduce
2011-11-01 11:14:18,642 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7635192% processing row indexed 53049 > reduce
2011-11-01 11:14:22,149 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7673085% processing row indexed 55067 > reduce
2011-11-01 11:14:25,151 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7725345% processing row indexed 57980 > reduce
2011-11-01 11:14:28,153 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7775895% processing row indexed 60698 > reduce
2011-11-01 11:14:31,155 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.78247994% processing row indexed 63422 > reduce
2011-11-01 11:14:34,157 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7878741% processing row indexed 66340 > reduce
2011-11-01 11:14:37,158 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.7932137% processing row indexed 69314 > reduce
2011-11-01 11:14:40,160 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.79808366% processing row indexed 72034 > reduce
2011-11-01 11:14:43,893 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.80219805% processing row indexed 74327 > reduce
2011-11-01 11:14:46,905 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8075403% processing row indexed 77215 > reduce
2011-11-01 11:14:49,936 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8122639% processing row indexed 79854 > reduce
2011-11-01 11:14:52,954 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.81731737% processing row indexed 82586 > reduce
2011-11-01 11:14:55,956 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.82233346% processing row indexed 85389 > reduce
2011-11-01 11:14:58,963 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8274955% processing row indexed 88191 > reduce
2011-11-01 11:15:01,965 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8325006% processing row indexed 90984 > reduce
2011-11-01 11:15:04,967 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8375456% processing row indexed 93732 > reduce
2011-11-01 11:15:07,969 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.84251565% processing row indexed 96504 > reduce
2011-11-01 11:15:10,971 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.84486747% processing row indexed 97810 > reduce
2011-11-01 11:15:13,976 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.849915% processing row indexed 100580 > reduce
2011-11-01 11:15:16,978 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8550106% processing row indexed 103380 > reduce
2011-11-01 11:15:19,980 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8603591% processing row indexed 106332 > reduce
2011-11-01 11:15:22,981 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8652568% processing row indexed 109054 > reduce
2011-11-01 11:15:25,983 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8705907% processing row indexed 111995 > reduce
2011-11-01 11:15:32,347 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8739246% processing row indexed 113831 > reduce
2011-11-01 11:15:35,349 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201110251936_0044_r_000005_0 0.8739246% processing row indexed 113831 > reduce
2011-11-01 11:17:44,077 WARN org.apache.hadoop.mapred.DefaultTaskController: Exit code from task is : 1
2011-11-01 11:17:44,081 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201110251936_0044_r_-1341282920 exited with exit code 1. Number of tasks it ran: 0
2011-11-01 11:17:44,082 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201110251936_0044_r_000005_0 : Child Error
java.io.IOException: Task process exit with nonzero status of 1.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:229)
2011-11-01 11:17:47,083 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 1
2011-11-01 11:17:47,247 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201110251936_0044_r_000005_0 task's state:FAILED_UNCLEAN
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment