Created
September 21, 2015 23:20
-
-
Save Aslan/9c7abfd063c94142a5ca to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| scala> textFile.count() | |
| 15/09/21 23:19:17 INFO FileInputFormat: Total input paths to process : 1 | |
| 15/09/21 23:19:17 INFO SparkContext: Starting job: count at <console>:24 | |
| 15/09/21 23:19:18 INFO DAGScheduler: Got job 0 (count at <console>:24) with 2 output partitions (allowLocal=false) | |
| 15/09/21 23:19:18 INFO DAGScheduler: Final stage: ResultStage 0(count at <console>:24) | |
| 15/09/21 23:19:18 INFO DAGScheduler: Parents of final stage: List() | |
| 15/09/21 23:19:18 INFO DAGScheduler: Missing parents: List() | |
| 15/09/21 23:19:18 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at textFile at <console>:21), which has no missing parents | |
| 15/09/21 23:19:18 INFO MemoryStore: ensureFreeSpace(2960) called with curMem=116570, maxMem=278302556 | |
| 15/09/21 23:19:18 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 265.3 MB) | |
| 15/09/21 23:19:18 INFO MemoryStore: ensureFreeSpace(1745) called with curMem=119530, maxMem=278302556 | |
| 15/09/21 23:19:18 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1745.0 B, free 265.3 MB) | |
| 15/09/21 23:19:18 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 10.0.6.27:45011 (size: 1745.0 B, free: 265.4 MB) | |
| 15/09/21 23:19:18 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:874 | |
| 15/09/21 23:19:18 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at textFile at <console>:21) | |
| 15/09/21 23:19:18 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks | |
| 15/09/21 23:19:33 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources | |
| 15/09/21 23:19:40 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, ip-10-0-2-7.ec2.internal, PROCESS_LOCAL, 1403 bytes) | |
| 15/09/21 23:19:40 INFO TaskSetManager: Re-queueing tasks for 20150828-232006-1225064458-5050-1694-S2 from TaskSet 0.0 | |
| 15/09/21 23:19:40 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, ip-10-0-2-7.ec2.internal): ExecutorLostFailure (executor 20150828-232006-1225064458-5050-1694-S2 lost) | |
| 15/09/21 23:19:41 INFO DAGScheduler: Executor lost: 20150828-232006-1225064458-5050-1694-S2 (epoch 0) | |
| 15/09/21 23:19:41 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 1, ip-10-0-2-7.ec2.internal, PROCESS_LOCAL, 1403 bytes) | |
| 15/09/21 23:19:41 INFO BlockManagerMasterEndpoint: Trying to remove executor 20150828-232006-1225064458-5050-1694-S2 from BlockManagerMaster. | |
| 15/09/21 23:19:41 INFO BlockManagerMaster: Removed 20150828-232006-1225064458-5050-1694-S2 successfully in removeExecutor | |
| 15/09/21 23:19:41 INFO DAGScheduler: Host added was in lost list earlier: ip-10-0-2-7.ec2.internal | |
| 15/09/21 23:19:41 INFO TaskSetManager: Re-queueing tasks for 20150828-232006-1225064458-5050-1694-S2 from TaskSet 0.0 | |
| 15/09/21 23:19:41 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1, ip-10-0-2-7.ec2.internal): ExecutorLostFailure (executor 20150828-232006-1225064458-5050-1694-S2 lost) | |
| 15/09/21 23:19:41 INFO DAGScheduler: Executor lost: 20150828-232006-1225064458-5050-1694-S2 (epoch 1) | |
| 15/09/21 23:19:41 INFO BlockManagerMasterEndpoint: Trying to remove executor 20150828-232006-1225064458-5050-1694-S2 from BlockManagerMaster. | |
| 15/09/21 23:19:41 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 2, ip-10-0-2-7.ec2.internal, PROCESS_LOCAL, 1403 bytes) | |
| 15/09/21 23:19:41 INFO BlockManagerMaster: Removed 20150828-232006-1225064458-5050-1694-S2 successfully in removeExecutor | |
| 15/09/21 23:19:41 INFO DAGScheduler: Host added was in lost list earlier: ip-10-0-2-7.ec2.internal | |
| 15/09/21 23:19:41 INFO TaskSetManager: Re-queueing tasks for 20150828-232006-1225064458-5050-1694-S2 from TaskSet 0.0 | |
| 15/09/21 23:19:41 WARN TaskSetManager: Lost task 0.2 in stage 0.0 (TID 2, ip-10-0-2-7.ec2.internal): ExecutorLostFailure (executor 20150828-232006-1225064458-5050-1694-S2 lost) | |
| 15/09/21 23:19:41 INFO DAGScheduler: Executor lost: 20150828-232006-1225064458-5050-1694-S2 (epoch 2) | |
| 15/09/21 23:19:41 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 3, ip-10-0-2-7.ec2.internal, PROCESS_LOCAL, 1403 bytes) | |
| 15/09/21 23:19:41 INFO BlockManagerMasterEndpoint: Trying to remove executor 20150828-232006-1225064458-5050-1694-S2 from BlockManagerMaster. | |
| 15/09/21 23:19:41 INFO BlockManagerMaster: Removed 20150828-232006-1225064458-5050-1694-S2 successfully in removeExecutor | |
| 15/09/21 23:19:41 INFO DAGScheduler: Host added was in lost list earlier: ip-10-0-2-7.ec2.internal | |
| 15/09/21 23:19:41 INFO TaskSetManager: Re-queueing tasks for 20150828-232006-1225064458-5050-1694-S2 from TaskSet 0.0 | |
| 15/09/21 23:19:41 WARN TaskSetManager: Lost task 0.3 in stage 0.0 (TID 3, ip-10-0-2-7.ec2.internal): ExecutorLostFailure (executor 20150828-232006-1225064458-5050-1694-S2 lost) | |
| 15/09/21 23:19:41 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job | |
| 15/09/21 23:19:41 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool | |
| 15/09/21 23:19:41 INFO TaskSchedulerImpl: Cancelling stage 0 | |
| 15/09/21 23:19:41 INFO DAGScheduler: ResultStage 0 (count at <console>:24) failed in 23.018 s | |
| 15/09/21 23:19:41 INFO DAGScheduler: Job 0 failed: count at <console>:24, took 23.348416 s | |
| 15/09/21 23:19:41 INFO DAGScheduler: Executor lost: 20150828-232006-1225064458-5050-1694-S2 (epoch 3) | |
| 15/09/21 23:19:41 INFO BlockManagerMasterEndpoint: Trying to remove executor 20150828-232006-1225064458-5050-1694-S2 from BlockManagerMaster. | |
| 15/09/21 23:19:41 INFO BlockManagerMaster: Removed 20150828-232006-1225064458-5050-1694-S2 successfully in removeExecutor | |
| 15/09/21 23:19:41 INFO DAGScheduler: Host added was in lost list earlier: ip-10-0-2-7.ec2.internal | |
| org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, ip-10-0-2-7.ec2.internal): ExecutorLostFailure (executor 20150828-232006-1225064458-5050-1694-S2 lost) | |
| Driver stacktrace: | |
| at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1273) | |
| at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1264) | |
| at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1263) | |
| at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) | |
| at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) | |
| at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1263) | |
| at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730) | |
| at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:730) | |
| at scala.Option.foreach(Option.scala:236) | |
| at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:730) | |
| at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1457) | |
| at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1418) | |
| at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment