join-on-sparkshell-info.log

Vertex failed, vertexName=0, vertexId=vertex_1417036912823_0059_1_02, diagnostics=[Task failed, taskId=task_1417036912823_0059_1_02_000014, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.IllegalStateException: Failed to execute processor for Vertex 2
at org.apache.spark.tez.SparkTaskProcessor.run(SparkTaskProcessor.scala:55)
at org.apache.tez.runtime.library.processor.SimpleProcessor.run(SimpleProcessor.java:53)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: java.io.IOException: All datanodes 10.240.9.169:50010 are bad. Aborting...
at org.apache.spark.tez.VertexResultTask.runTask(VertexResultTask.scala:93)
at org.apache.spark.tez.SparkUtils$.runTask(SparkUtils.scala:146)
at org.apache.spark.tez.SparkTaskProcessor.doRun(SparkTaskProcessor.scala:91)
at org.apache.spark.tez.SparkTaskProcessor.run(SparkTaskProcessor.scala:51)
... 13 more
Caused by: java.io.IOException: All datanodes 10.240.9.169:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1137)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:933)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:487)
]], Vertex failed as one or more tasks failed. failedTasks:1, Vertex vertex_1417036912823_0059_1_02 [0] killed/failed due to:null]
DAG failed due to vertex failure. failedVertices:1 killedVertices:0

join-on-sparkshell.log

14/11/28 04:09:32 INFO client.DAGClientImpl: DAG: State: FAILED Progress: 39.13% TotalTasks: 46 Succeeded: 18 Running: 0 Failed: 4 Killed: 24 FailedTaskAttempts: 4
14/11/28 04:09:32 INFO client.DAGClientImpl:    VertexStatus: VertexName: 1 Progress: 100% TotalTasks: 9 Succeeded: 9 Running: 0 Failed: 0 Killed: 0
14/11/28 04:09:32 INFO client.DAGClientImpl:    VertexStatus: VertexName: 2 Progress: 100% TotalTasks: 9 Succeeded: 9 Running: 0 Failed: 0 Killed: 0
14/11/28 04:09:32 INFO client.DAGClientImpl:    VertexStatus: VertexName: 0 Progress: 0% TotalTasks: 28 Succeeded: 0 Running: 0 Failed: 4 Killed: 24 FailedTaskAttempts: 4
14/11/28 04:09:32 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
14/11/28 04:09:32 INFO tez.DAGBuilder: DAG execution complete
14/11/28 04:09:32 ERROR tez.DAGBuilder: DAG diagnostics: [Vertex failed, vertexName=0, vertexId=vertex_1417036912823_0056_1_02, diagnostics=[Task failed, taskId=task_1417036912823_0056_1_02_000007, diagnostics=[TaskAttempt 0 failed, info=[AttemptID:attempt_1417036912823_0056_1_02_000007_0 Timed out after 300 secs]], Task failed, taskId=task_1417036912823_0056_1_02_000003, diagnostics=[TaskAttempt 0 failed, info=[AttemptID:attempt_1417036912823_0056_1_02_000003_0 Timed out after 300 secs]], Task failed, taskId=task_1417036912823_0056_1_02_000011, diagnostics=[TaskAttempt 0 failed, info=[AttemptID:attempt_1417036912823_0056_1_02_000011_0 Timed out after 300 secs]], Task failed, taskId=task_1417036912823_0056_1_02_000008, diagnostics=[TaskAttempt 0 failed, info=[AttemptID:attempt_1417036912823_0056_1_02_000008_0 Timed out after 300 secs]], Vertex failed as one or more tasks failed. failedTasks:4, Vertex vertex_1417036912823_0056_1_02 [0] killed/failed due to:null], DAG failed due to vertex failure. failedVertices:1 killedVertices:0]