Created
November 4, 2014 21:58
-
-
Save shrijeet/6cc7acf39cae55045f55 to your computer and use it in GitHub Desktop.
MAPREDUCE-4815 Failing pi job
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[shrijeet@xxx-209 ~]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 5 5 | |
Number of Maps = 5 | |
Samples per Map = 5 | |
Wrote input for Map #0 | |
Wrote input for Map #1 | |
Wrote input for Map #2 | |
Wrote input for Map #3 | |
Wrote input for Map #4 | |
Starting Job | |
14/11/04 16:53:51 INFO client.RMProxy: Connecting to ResourceManager at yarn-rm.xxx-yyyy.example.net/172.0.0.0:8032 | |
14/11/04 16:53:52 INFO input.FileInputFormat: Total input paths to process : 5 | |
14/11/04 16:53:52 INFO mapreduce.JobSubmitter: number of splits:5 | |
14/11/04 16:53:53 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1415051647016_0018 | |
14/11/04 16:53:53 INFO impl.YarnClientImpl: Submitted application application_1415051647016_0018 | |
14/11/04 16:53:53 INFO mapreduce.Job: The url to track the job: http://yarn-rm.xxx-yyyy.example.net:8088/proxy/application_1415051647016_0018/ | |
14/11/04 16:53:53 INFO mapreduce.Job: Running job: job_1415051647016_0018 | |
14/11/04 16:54:00 INFO mapreduce.Job: Job job_1415051647016_0018 running in uber mode : false | |
14/11/04 16:54:00 INFO mapreduce.Job: map 0% reduce 0% | |
14/11/04 16:54:08 INFO mapreduce.Job: map 100% reduce 0% | |
14/11/04 16:54:20 INFO mapreduce.Job: map 100% reduce 100% | |
14/11/04 16:54:23 INFO mapreduce.Job: Job job_1415051647016_0018 completed successfully | |
14/11/04 16:54:24 INFO mapreduce.Job: Counters: 49 | |
File System Counters | |
FILE: Number of bytes read=46 | |
FILE: Number of bytes written=632867 | |
FILE: Number of read operations=0 | |
FILE: Number of large read operations=0 | |
FILE: Number of write operations=0 | |
HDFS: Number of bytes read=1345 | |
HDFS: Number of bytes written=215 | |
HDFS: Number of read operations=27 | |
HDFS: Number of large read operations=0 | |
HDFS: Number of write operations=3 | |
Job Counters | |
Launched map tasks=5 | |
Launched reduce tasks=1 | |
Data-local map tasks=5 | |
Total time spent by all maps in occupied slots (ms)=56728 | |
Total time spent by all reduces in occupied slots (ms)=17810 | |
Total time spent by all map tasks (ms)=28364 | |
Total time spent by all reduce tasks (ms)=8905 | |
Total vcore-seconds taken by all map tasks=28364 | |
Total vcore-seconds taken by all reduce tasks=8905 | |
Total megabyte-seconds taken by all map tasks=72611840 | |
Total megabyte-seconds taken by all reduce tasks=22796800 | |
Map-Reduce Framework | |
Map input records=5 | |
Map output records=10 | |
Map output bytes=90 | |
Map output materialized bytes=122 | |
Input split bytes=755 | |
Combine input records=0 | |
Combine output records=0 | |
Reduce input groups=2 | |
Reduce shuffle bytes=122 | |
Reduce input records=10 | |
Reduce output records=0 | |
Spilled Records=20 | |
Shuffled Maps =5 | |
Failed Shuffles=0 | |
Merged Map outputs=5 | |
GC time elapsed (ms)=47 | |
CPU time spent (ms)=7250 | |
Physical memory (bytes) snapshot=3494408192 | |
Virtual memory (bytes) snapshot=17816207360 | |
Total committed heap usage (bytes)=6068109312 | |
Shuffle Errors | |
BAD_ID=0 | |
CONNECTION=0 | |
IO_ERROR=0 | |
WRONG_LENGTH=0 | |
WRONG_MAP=0 | |
WRONG_REDUCE=0 | |
File Input Format Counters | |
Bytes Read=590 | |
File Output Format Counters | |
Bytes Written=97 | |
Job Finished in 33.23 seconds | |
java.io.FileNotFoundException: File does not exist: hdfs://xxx-yyyy-zz:8020/user/shrijeet/QuasiMonteCarlo_1415138028494_861882293/out/reduce-out | |
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) | |
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) | |
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) | |
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) | |
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1749) | |
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773) | |
at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314) | |
at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354) | |
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) | |
at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363) | |
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) | |
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | |
at java.lang.reflect.Method.invoke(Method.java:597) | |
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72) | |
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145) | |
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74) | |
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) | |
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) | |
at java.lang.reflect.Method.invoke(Method.java:597) | |
at org.apache.hadoop.util.RunJar.main(RunJar.java:212) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment